  An interesting test of Einstein's equivalence principle (EEP) relies on the
observed lag in arrival times of photons emitted from extragalactic transient
sources. Attributing the lag between photons of different energies to the
gravitational potential of the Milky Way (MW), several authors derive new
constraints on deviations from EEP. It is shown here that potential
fluctuations from the large scale structure are at least two orders of
magnitude larger than the gravitational potential of the MW. Combined with the
larger distances, for sources at redshift $z\gtsim 0.5$ the {\it rms} of the
contribution from these fluctuations exceeds the MW by more than 4 orders of
magnitude. We provide actual constraints for several objects based on a
statistical calculation of the large scale fluctuations in the standard
$\Lambda$CDM cosmological model.
