  U-duality plays a special role in the study of the microscopic degrees of
freedom of supersymmetric black holes. To be consistent with duality, the black
hole quantum degeneracy must obey special arithmetic properties, which are
non-perturbative in nature. In this work, we study these properties from a
holographic point of view, establishing a connection between arithmetic
properties of Kloosterman sums and quantum gravity in $\text{AdS}_2$ space. To
this end, we consider the entropy of black holes that carry non-primitive
charges, in both $\mathcal{N}=8$ and $\mathcal{N}=4$ four dimensional
compactifications; our analysis includes all the perturbative and
non-perturbative bulk quantum corrections. The key result relies on special
arithmetic properties of generalized Kloosterman sums that we develop. These
are a generalization of the known Selberg identity of classical Kloosterman
sums. In both the $\mathcal{N}=8$ and $\mathcal{N}=4$ examples, we recover,
from the bulk quantum gravity, the non-primitive answer which is a sum over the
primitive degeneracies, depending non-trivially on the discrete duality
invariants. In particular, for the $\mathcal{N}=4$ case we show that the
quantum gravity answer reproduces the dependence on the torsion invariant
$I=\text{gcd}(Q\wedge P)$, in agreement with the microscopic formulas. For the
$\mathcal{N}=8$ case, we solve a puzzle related to U-duality invariance of the
supergravity answer and the corresponding one-eighth BPS degeneracy.
