  We investigate theoretically the quantum phase transition (QPT) between the
one-channel Kondo (1CK) and two-channel Kondo (2CK) fixed points in a quantum
dot coupled to helical edge states of interacting 2D topological insulators
(2DTI) with Luttinger parameter $0<K<1$. The model has been studied in Ref. 21,
and was mapped onto an anisotropic two-channel Kondo model via bosonization.
For K<1, the strong coupling 2CK fixed point was argued to be stable for
infinitesimally weak tunnelings between dot and the 2DTI based on a simple
scaling dimensional analysis[21]. We re-examine this model beyond the bare
scaling dimension analysis via a 1-loop renormalization group (RG) approach
combined with bosonization and re-fermionization techniques near weak-coupling
and strong-coupling (2CK) fixed points. We find for K -->1 that the 2CK fixed
point can be unstable towards the 1CK fixed point and the system may undergo a
quantum phase transition between 1CK and 2CK fixed points. The QPT in our model
comes as a result of the combined Kondo and the helical Luttinger physics in
2DTI, and it serves as the first example of the 1CK-2CK QPT that is accessible
by the controlled RG approach. We extract quantum critical and crossover
behaviors from various thermodynamical quantities near the transition. Our
results are robust against particle-hole asymmetry for 1/2<K<1.
