  Robustness of helical edge states in 2D topological insulators (TI) against
strong interactions remains an intriguing issue. Here, by performing the first
sign-free quantum Monte Carlo (QMC) simulation of the Kane-Mele-Hubbard-Rashba
model which describes an interacting 2D TI with two-particle backscattering on
edges, we verify that the gapless helical edge states are robust against a
finite range of two-particle backscattering when the Coulomb repulsion is not
strong. However, when the Coulomb repulsion is strong enough, the helical edge
states can be gapped by infinitesimal two-particle backscattering, resulting in
edge magnetic order. We further reveal universal properties of the magnetic
edge quantum critical point (EQCP). At magnetic domain walls on edges, we find
that a fractionalized charge of e/2 emerges. Implications of our results to
recent transport experiments in the InAs/GaSb quantum well, which is a 2D TI
with strong interactions, will also be discussed.
