  We study the four-point function of the lowest-lying half-BPS operators in
the ${\cal N} =4$ $SU(N)$ super-Yang-Mills theory and its relation to the
flat-space four-graviton amplitude in type IIB superstring theory. We work in a
large-$N$ expansion in which the complexified Yang-Mills coupling $\tau$ is
fixed. In this expansion, non-perturbative instanton contributions are present,
and the $SL(2, \mathbb{Z})$ duality invariance of correlation functions is
manifest. Our results are based on a detailed analysis of the sphere partition
function of the mass-deformed SYM theory, which was previously computed using
supersymmetric localization. This partition function determines a certain
integrated correlator in the undeformed ${\cal N} = 4$ SYM theory, which in
turn constrains the four-point correlator at separated points. In a
normalization where the two-point functions are proportional to $N^2-1$ and are
independent of $\tau$ and $\bar \tau$, we find that the terms of order
$\sqrt{N}$ and $1/\sqrt{N}$ in the large $N$ expansion of the four-point
correlator are proportional to the non-holomorphic Eisenstein series
$E({\scriptstyle \frac{3}{2}},\tau,\bar\tau)$ and $E({\scriptstyle
\frac{5}{2}},\tau,\bar\tau)$, respectively. In the flat space limit, these
terms match the corresponding terms in the type IIB S-matrix arising from $R^4$
and $D^4 R^4$ contact interactions, which, for the $R^4$ case, represents a
check of AdS/CFT at finite string coupling. Furthermore, we present striking
evidence that these results generalize so that, at order $N^{\frac{1}{2}-m}$
with integer $m \ge 0$, the expansion of the integrated correlator we study is
a linear sum of non-holomorphic Eisenstein series with half-integer index,
which are manifestly $SL(2,\mathbb{Z})$ invariant.
