  We develop a new approach to extracting the physical consequences of
S-duality of four-dimensional $\mathcal{N}=4$ super Yang-Mills (SYM) and its
string theory dual, based on $SL(2,\mathbb{Z})$ spectral theory. We observe
that CFT observables $\mathcal{O}$, invariant under $SL(2,\mathbb{Z})$
transformations of a complexified gauge coupling $\tau$, admit a unique
spectral decomposition into a basis of square-integrable functions. This
formulation has direct implications for the analytic structure of
$\mathcal{N}=4$ SYM data, both perturbatively and non-perturbatively in all
parameters. For example, $k$-instanton sectors are uniquely determined by the
zero- and one-instanton sectors, and Borel summable series around
$k$-instantons have convergence radii with simple $k$-dependence. In large $N$
limits, we derive the existence and scaling of non-perturbative effects, which
we exhibit for certain $\mathcal{N}=4$ SYM observables. An elegant benchmark
for these techniques is the integrated four-point function conjecturally
determined by [arXiv:2102.09537] for all $\tau$ for $SU(N)$ gauge group; we
derive and elucidate its form.
  These results have ramifications for holography. We explain how
$\langle\mathcal{O}\rangle$, the ensemble average over the $\mathcal{N}=4$
supersymmetric conformal manifold with respect to the Zamolodchikov measure, is
cleanly isolated by the spectral decomposition. We prove that the large $N$
limit of $\langle\mathcal{O}\rangle$ equals the large $N$, large 't Hooft
coupling limit of $\mathcal{O}$. Holographically speaking,
$\langle\mathcal{O}\rangle = \mathcal{O}_{\rm sugra}$, its value in type IIB
supergravity on AdS$_5 \times$ S$^5$. This result, which extends to all orders
in $1/N$, embeds ensemble averaging into the traditional AdS/CFT paradigm. The
statistics of the $SL(2,\mathbb{Z})$ ensemble exhibit both perturbative and
non-perturbative $1/N$ effects.
