  Two spins located at the edge of a quantum spin Hall insulator may interact
with each other via indirect spin-exchange interaction mediated by the helical
edge states, namely the RKKY interaction, which can be measured by the magnetic
correlation between the two spins. By means of the newly developed natural
orbitals renormalization group (NORG) method, we investigated the magnetic
correlation between two Kondo impurities interacting with the helical edge
states, based on the Kane-Mele model defined in a finite zigzag graphene
nanoribbon with spin-orbital coupling (SOC). We find that the SOC effect breaks
the symmetry in spatial distribution of the magnetic correlation, leading to
anisotropy in the RKKY interaction. Specifically, the total correlation is
always ferromagnetic (FM) when the two impurities are located at the same
sublattice, while it is always antiferromagnetic (AFM) when at the different
sublattices. Meanwhile, the behavior of the in-plane correlation is consistent
with that of the total correlation. However, the out-of-plane correlation can
be tuned from FM to AFM by manipulating either the Kondo coupling or the
interimpurity distance. Furthermore, the magnetic correlation is tunable by the
SOC, especially that the out-of-plane correlation can be adjusted from FM to
AFM by increasing the strength of SOC. Dynamic properties of the system,
represented by the spin-staggered excitation spectrum and the spin-staggered
susceptibility at the two impurity sites, are finally explored. It is shown
that the spin-staggered susceptibility is larger when the two impurities are
located at the different sublattices than at the same sublattice, which is
consistent with the behavior of the out-of-plane correlation. On the other
hand, our study further demonstrates that the NORG is an effective numerical
method for studying the quantum impurity systems.
