  We study the phase diagram of a model quantum spin Hall system as a function
of band inversion and band-coupling strength, demonstrating that when band
hybridization is weak, an interaction-induced nematic insulator state emerges
over a wide range of band inversion. This property is a consequence of the
long-range Coulomb interaction, which favors interband phase coherence that is
weakly dependent on momentum and therefore frustrated by the single-particle
Hamiltonian at the band inversion point. For weak band hybridization,
interactions convert the continuous gap closing topological phase transition at
inversion into a pair of continuous phase transitions bounding a state with
broken time-reversal and rotational symmetries. At intermediate band
hybridization, the topological phase transition proceeds instead via a quantum
anomalous Hall insulator state, whereas at strong hybridization interactions
play no role. We comment on the implications of our findings for InAs/GaSb and
HgTe/CdTe quantum spin Hall systems.
