  We describe the finite-size spectrum in the vicinity of the quantum critical
point between a $\mathbb{Z}_2$ spin liquid and a coplanar antiferromagnet on
the torus. We obtain the universal evolution of all low-lying states in an
antiferromagnet with global SU(2) spin rotation symmetry, as it moves from the
4-fold topological degeneracy in a gapped $\mathbb{Z}_2$ spin liquid to the
Anderson "tower-of-states" in the ordered antiferromagnet. Due to the existence
of nontrivial order on either side of this transition, this critical point
cannot be described in a conventional Landau-Ginzburg-Wilson framework. Instead
it is described by a theory involving fractionalized degrees of freedom known
as the O$(4)^\ast$ model, whose spectrum is altered in a significant way by its
proximity to a topologically ordered phase. We compute the spectrum by relating
it to the spectrum of the O$(4)$ Wilson-Fisher fixed point on the torus,
modified with a selection rule on the states, and with nontrivial boundary
conditions corresponding to topological sectors in the spin liquid. The
spectrum of the critical O($2N$) model is calculated directly at $N=\infty$,
which then allows a reconstruction of the full spectrum of the O($2N)^\ast$
model at leading order in 1/N. This spectrum is a unique characteristic of the
vicinity of a fractionalized quantum critical point, as well as a universal
signature of the existence of proximate $\mathbb{Z}_2$ topological and
antiferromagnetically-ordered phases, and can be compared with numerical
computations on quantum antiferromagnets on two dimensional lattices.
