  We systematically explore the ground-state phase diagram of the
Kane-Mele-Kondo lattice model on the honeycomb lattice, in particular, we focus
on its magnetic properties which has not been studied in the previous
publication[Feng, Dai, Chung, and Si, Phys. Rev. Lett. \textbf{111}, 016402
(2013)]. Beside the Kondo insulator found in that paper, two kinds of
antiferromagnetic spin-density-wave phases are identified. One is the normal
antiferromagnetic spin-density-wave state and the other is a nontrivial
topological antiferromagnetic spin-density-wave state with a quantized spin
Hall conductance and a helical edge-state. The quantum spin Hall insulator is
found to be absent since it is always unstable to antiferromagnetic
spin-density-wave states at least at the mean-field level in our model.
Furthermore, the transition between the two spin-density-wave phases are
topological quantum phase transition described by the three-dimensional quantum
electrodynamics, in which conduction electrons contribute to the low-energy
Dirac fermions while the spin-wave fluctuation of local spins gives rise to an
effective dynamic U(1) gauge-field. Such nontrivial transition shows radical
critical thermodynamic, transport and single-particle behaviors, which provide
a fingerprint for this transition. Additionally, the transition of
antiferromagnetic spin-density-wave states to the Kondo insulator is found to
be first-order. The introduction of two novel magnetic phases and their
topological quantum phase transition show rich and intrinsic physics involving
in the Kane-Mele-Kondo lattice model.
