  We study modular invariants arising in the four-point functions of the stress
tensor multiplet operators of the ${\cal N} = 4$ $SU(N)$ super-Yang-Mills
theory, in the limit where $N$ is taken to be large while the complexified
Yang-Mills coupling $\tau$ is held fixed. The specific four-point functions we
consider are integrated correlators obtained by taking various combinations of
four derivatives of the squashed sphere partition function of the ${\cal N} =
2^*$ theory with respect to the squashing parameter $b$ and mass parameter $m$,
evaluated at the values $b=1$ and $m=0$ that correspond to the ${\cal N} = 4$
theory on a round sphere. At each order in the $1/N$ expansion, these fourth
derivatives are modular invariant functions of $(\tau, \bar \tau)$. We present
evidence that at half-integer orders in $1/N$, these modular invariants are
linear combinations of non-holomorphic Eisenstein series, while at integer
orders in $1/N$, they are certain "generalized Eisenstein series" which satisfy
inhomogeneous Laplace eigenvalue equations on the hyperbolic plane. These
results reproduce known features of the low-energy expansion of the
four-graviton amplitude in type IIB superstring theory in ten-dimensional flat
space and have interesting implications for the structure of the analogous
expansion in $AdS_5\times S^5$.
