  For any given network of detectors, and for any given integration time, even
in the idealized limit of negligible instrumental noise, the intrinsic time
variation of the isotropic component of the Stochastic Gravitational Wave
Background (SGWB) induces a limit on how accurately the anisotropies in the
SGWB can be measured. We show here how this sample limit can be calculated and
apply this to three separate configurations of ground-based detectors placed at
existing and planned sites. Our results show that in the idealized, best-case
scenario individual multipoles of the anisotropies at $\ell \leq 8$ can only be
measured to $\sim 10^{-5} - 10^{-4}$ level over 5 years of observation as a
fraction of the isotropic component. As the sensitivity improves as the square
root of the observation time, this poses a very serious challenge for the
measurement of the anisotropies of SGWB of cosmological origin, even in the
case of idealised detectors with arbitrarily low instrumental noise.
