  We study the RKKY interaction mediated by the helical edge states of a
quantum spin Hall insulator in the presence of the Rashba spin-orbital coupling
induced by an external electric field and the electron-electron interaction. We
show that in the presence of the Rashba coupling, the RKKY interaction induced
by the helical edge states contains not only the Heisenberg-like and the
Dzyaloshinskii-Moria terms but also the nematic-type term that is not present
originally, with the range functions depending on the strength of the Rashba
coupling. We also show that the electron-electron interaction changes the
strength of the RKKY interaction by modifying the power of the $1/|x|$
dependence of the range functions. In particular, by varying the strength of
the interaction or the Rashba coupling, there is an (impurity) quantum phase
transition involving the sign change of the RKKY interaction at the value of
the Luttinger liquid parameter $K=1/2$. Since both the strength of the Rashba
coupling and the chemical potential of the helical edge states are electrically
controllable by external gate voltages, our results not only shed light on the
nature of magnetic impurity correlations in the edge of a two-dimensional
topological insulator, but also pave a way to manipulate the qubits in quantum
computing.
