  We study Heavy-Heavy-Light-Light (HHLL) correlators $\langle \mathcal{H}
\mathcal{H} \mathcal{O}_2 \mathcal{O}_2 \rangle$ in $\mathcal{N}=4$ super
Yang-Mills theory with $SU(N)$ gauge group at generic $N$. The light operator
$\mathcal{O}_2$ is the dimension two superconformal primary in the stress
tensor multiplet and $\mathcal{H}$ is a general half-BPS superconformal primary
operator with dimension (or $R$-charge) $\Delta_{\mathcal{H}}$. We consider the
large-charge 't Hooft limit, where $\Delta_{\mathcal{H}} \rightarrow \infty$
with fixed 't Hooft-like coupling $\lambda:=\Delta_{\mathcal{H}}\, g_{_{\rm
YM}}^2$. We show that the $L$-loop contribution to the HHLL correlators in the
leading large-charge limit is universal for any choice of the heavy operator
$\mathcal{H}$, given as $\lambda^L \sum_{\ell=0}^{L} \Phi^{(\ell)}
\Phi^{(L-\ell)}$ with an $SU(N)$ colour factor coefficient, where
$\Phi^{(\ell)}$ is the ladder Feynman integral, which is known to all loops.
The dependence on the explicit form of the heavy operator lies only in the
colour factor coefficients. We determine such colour factors for several
classes of heavy operators, and show that the large charge limit leads to
minimal powers of $N$. For the special class of "canonical heavy operators",
one can even resum the all-loop ladder integrals and determine the correlators
at finite $\lambda$. Furthermore, upon integrating over the spacetime
dependence, the resulting integrated HHLL correlators agree with the existing
results derived from supersymmetric localisation. Finally, as an application of
the all-loop analytic results, we derive exact expressions for the structure
constants of two heavy operators and the Konishi operator, finding intriguing
connections with the integrated HHLL correlators.
