  The Hilbert space of a quantum system with internal global symmetry $G$
decomposes into sectors labelled by irreducible representations of $G$. If the
system is chaotic, the energies in each sector should separately resemble
ordinary random matrix theory. We show that such "sector-wise" random matrix
ensembles arise as the boundary dual of two-dimensional gravity with a $G$
gauge field in the bulk. Within each sector, the eigenvalue density is enhanced
by a nontrivial factor of the dimension of the representation, and the ground
state energy is determined by the quadratic Casimir. We study the consequences
of 't Hooft anomalies in the matrix ensembles, which are incorporated by adding
specific topological terms to the gauge theory action. The effect is to
introduce projective representations into the decomposition of the Hilbert
space. Finally, we consider ensembles with $G$ symmetry and time reversal
symmetry, and analyze a simple case of a mixed anomaly between time reversal
and an internal $\mathbb{Z}_2$ symmetry.
