  Considering the matter power spectrum covariance matrix, it has recently been
found that there is a potentially dominant effect on mildly non-linear scales
due to power in modes of size equal to and larger than the survey volume. This
{\it beat coupling} effect has been derived analytically in perturbation theory
and while it has been tested with simulations, some questions remain
unanswered. Moreover, there is an additional effect of these large modes, which
has so far not been included in analytic studies, namely the effect on the
estimated {\it average} density which enters the power spectrum estimate. In
this article, we work out analytic, perturbation theory based expressions
including both the beat coupling and this {\it local average effect} and we
show that while, when isolated, beat coupling indeed causes large excess
covariance in agreement with the literature, in a realistic scenario this is
compensated almost entirely by the local average effect, leaving only $\sim 10
%$ of the excess. We test our analytic expressions by comparison to a suite of
large N-body simulations. For the variances, we find excellent agreement with
the analytic expressions for $k < 0.2 h$Mpc$^{-1}$ at $z=0.5$, while the
correlation coefficients agree to beyond $k=0.4 h$Mpc$^{-1}$. As expected, the
range of agreement increases towards higher redshift and decreases slightly
towards $z=0$. We finish by including the large-mode effects in a full
covariance matrix description for arbitrary survey geometry and confirming its
validity using simulations. This may be useful as a stepping stone towards
building an actual galaxy (or other tracer's) power spectrum covariance matrix.
[abridged]
