  Twisted transition metal dichalcogenide (TMD) homobilayers have recently
emerged as a powerful platform for studying correlated insulating states. In
the strongly correlated limit, we construct an effective spin Hamiltonian on a
honeycomb lattice that includes the Heisenberg interaction and nonsymmetric
interactions such as a Dzyaloshinskii-Moriya interaction and a Kane-Mele
coupling for the Mott-insulating phase at half-filling. For the twisted TMD
homobilayers, the spin-orbit coupling in the Hubbard model, which is expected
to induce the antisymmetric exchange couplings in the effective spin
Hamiltonian, is a highly tunable and experimentally accessible quantity that
can be tuned by an applied electric field. In this study, we investigate
classical and quantum phase diagrams of the effective spin Hamiltonian using
analytical and numerical methods. We show that the model exhibits a rich
classical phase diagram including an antiferromagnetic (AFM) phase, a planar
spiral ordered phase with high classical degeneracy, a $z$-AFM phase, a
noncoplanar phase, a noncollinear phase, and a 120$^{\circ}$-AFM phase. In the
quantum treatment, we calculate low-energy magnon excitation spectrum, ground
state energy, and static spin structure factor using linear spin-wave theory
and density matrix renormalization group methods to compose the quantum phase
diagram of the effective spin Hamiltonian. Beyond the Heisenberg interaction,
we find that the existence of these antisymmetric couplings is responsible for
the quantum spin liquid, $z$-AFM, noncoplanar, and 120$^{\circ}$ phases.
Twisted TMD homobilayers, therefore, offer rich platforms for realizing rich
phases of matter such as quantum spin liquid, noncoplanar, and 120$^{\circ}$,
resulting from the spin-orbit coupling.
