  We study collective behavior of magnetic adatoms randomly distributed on the
surface of a topological insulator. As a consequence of the spin-momentum
locking on the surface, the RKKY-type interactions of two adatom spins depend
on the direction of the vector connecting them, thus interactions of an
ensemble of adatoms are frustrated. We show that at low temperatures the
frustrated RKKY interactions give rise to two phases: an ordered ferromagnetic
phase with spins pointing perpendicular to the surface, and a disordered
spin-glass-like phase. The two phases are separated by a quantum phase
transition driven by the magnetic exchange anisotropy. Ferromagnetic ordering
occurs via a finite-temperature phase transition. The ordered phase breaks
time-reversal symmetry spontaneously, driving the surface states into a gapped
state, which exhibits an anomalous quantum Hall effect and provides a
realization of the parity anomaly. We find that the magnetic ordering is
suppressed by potential scattering. Our work indicates that controlled
deposition of magnetic impurities provides a way to modify the electronic
properties of topological insulators.
