  We investigate the Hubbard model on the honeycomb lattice with intrinsic spin
orbit interactions as a paradigm for two-dimensional topological band
insulators in the presence of interactions. Applying a combination of
Hartree-Fock theory, slave-rotor techniques, and topological arguments, we show
that the topological band insulating phase persists up to quite strong
interactions. Then we apply the slave-rotor mean-field theory and find a Mott
transition at which the charge degrees of freedom become localized on the
lattice sites. The spin degrees of freedom, however, are still described by the
original Kane-Mele band structure. Gauge field effects in this region play an
important role. When the honeycomb layer is isolated then the spin sector
becomes already unstable toward an easy plane Neel order. In contrast, if the
honeycomb lattice is surrounded by extra "screening" layers with gapless
spinons, then the system will support a fractionalized topological insulator
phase with gapless spinons at the edges. For large interactions, we derive an
effective spin Hamiltonian.
