  We study quantum corrections in the gravitational path integral around nearly
$1/16$-BPS black holes in asymptotically $AdS_5 \times S^5$ space, dual to
heavy states in 4D $\mathcal{N}=4$ super Yang-Mills. The analysis provides a
gravitational explanation of why $1/16$-BPS black holes exhibit an exact
degeneracy at large $N$ and why all such states have the same charges,
confirming the belief that the superconformal index precisely counts the
entropy of extremal black holes. We show the presence of a gap of order
$N^{-2}$ between the $1/16$-BPS black holes and the lightest near-BPS black
holes within the same charge sector. This is the first example of such a gap
for black holes states within the context of $AdS_5$ holography. We also derive
the spectrum of near-BPS states that lie above this gap. Our computation relies
on finding the correct version of the $\mathcal{N}=2$ super-Schwarzian theory
which captures the breaking of the $SU(1, 1|1)$ symmetry when the black hole
has finite temperature and non-zero chemical potential. Finally, we comment on
possible stringy and non-perturbative corrections that can affect the black
hole spectrum.
