  We perform a systematic study of integrated four-point functions of half-BPS
operators in four-dimensional $\mathcal{N}=4$ super Yang-Mills theory with
gauge group $SU(N)$. These observables, defined by a certain spacetime integral
of $\langle\mathcal{O}_2\mathcal{O}_2\mathcal{O}_p\mathcal{O}_p\rangle$ where
$\mathcal{O}_p$ is a superconformal primary of charge $p$, are known to be
computable by supersymmetric localization, yet are non-trivial functions of the
complexified gauge coupling $\tau$. We find explicit and remarkably simple
results for several classes of these observables, exactly as a function of $N$
and $\tau$. Their physical and formal properties are greatly illuminated upon
employing the $SL(2,\mathbb{Z})$ spectral decomposition: in this
S-duality-invariant eigenbasis, the integrated correlators are fixed simply by
polynomials in the spectral parameter. These polynomials are determined
recursively by linear algebraic equations relating different $N$ and $p$, such
that all integrated correlators are ultimately fixed in terms of the integrated
stress tensor multiplets in the $SU(2)$ theory. Our computations include the
full matrix of integrated correlators at low values of $p$, and a certain
infinite class involving operators of arbitrary $p$. The latter satisfy an open
lattice chain equation for all $N$, reminiscent of the Toda equation obeyed by
extremal correlators in $\mathcal{N}=2$ superconformal theories. We compute
ensemble averages of these observables and analyze our solutions at large $N$,
confirming and predicting features of semiclassical AdS$_5\, \times$ S$^5$
supergravity amplitudes.
