  Anyons in a topologically ordered phase can carry fractional quantum numbers
with respect to the symmetry group of the considered system, one example being
the fractional charge of the quasiparticles in the fractional quantum Hall
effect. When such symmetry-fractionalized anyons condense, the resulting phase
must spontaneously break the symmetry and display a local order parameter. In
this paper, we study the phase diagram and anyon condensation transitions of a
$\mathbb{Z}_2$ topological order perturbed by Ising interactions in the Toric
Code. The interplay between the global Ising symmetry and the lattice space
group symmetries results in a non-trivial symmetry fractionalization class for
the anyons, and is shown to lead to two characteristically different
symmetry-broken phases. To understand the anyon condensation transitions, we
use the recently introduced critical torus energy spectrum technique to
identify a line of emergent 2+1D XY* transitions ending at a fine-tuned
(Ising$^2$)* critical point. We provide numerical evidence for the occurrence
of two symmetry breaking patterns predicted by the specific symmetry
fractionalization class of the anyons in the explored phase diagram. In
combination with quantum Monte Carlo simulations we measure unusually large
critical exponents for the scaling of the correlation function at the anyon
condensation transitions, and we identify lines of (weakly) first order
transitions in the phase diagram. As an important result, we discuss the phase
diagram of a resulting 2+1D Ashkin-Teller model, where we demonstrate that
torus spectroscopy is capable of identifying emergent XY/O(2) critical
behaviour, thereby solving some longstanding open questions in the domain of
the 3D Ashkin-Teller models. To establish the generality of our results, we
propose a field theoretical description which is in excellent agreement with
the numerical results.
