  In the pursuit of primordial non-Gaussianities, we hope to access smaller
scales across larger comoving volumes. At low redshift, the search for
primordial non-Gaussianities is hindered by gravitational collapse, to which we
often associate a scale $k_{\rm NL}$. Beyond these scales, it will be hard to
reconstruct the modes sensitive to the primordial distribution. When
forecasting future constraints on the amplitude of primordial non-Gaussianity,
$f_{\rm NL}$, off-diagonal components are usually neglected in the covariance
because these are small compared to the diagonal. We show that the induced
non-Gaussian off-diagonal components in the covariance degrade forecast
constraints on primordial non-Gaussianity, even when all modes are well within
what is usually considered the linear regime. As a testing ground, we examine
the effects of these off-diagonal components on the constraining power of the
matter bispectrum on $f_{\rm NL}$ as a function of $k_{\rm max}$ and redshift,
confirming our results against N-body simulations out to redshift $z=10$. We
then consider these effects on the hydrogen bispectrum as observed from a
PUMA-like 21-cm intensity mapping survey at redshifts $2<z<6$ and show that not
including off-diagonal covariance over-predicts the constraining power on
$f_{\rm NL}$ by up to a factor of $5$. For future surveys targeting even higher
redshifts, such as Cosmic Dawn and the Dark Ages, which are considered ultimate
surveys for primordial non-Gaussianity, we predict that non-Gaussian covariance
would severely limit prospects to constrain $f_{\rm NL}$ from the bispectrum.
