  We study the effective spin-orbital model for honeycomb-layered transition
metal compounds, applying the second-order perturbation theory to the
three-orbital Hubbard model with the anisotropic hoppings. This model is
reduced to the Kitaev model in the strong spin-orbit coupling limit. Combining
the cluster mean-field approximations with the exact diagonalization, we treat
the Kugel-Khomskii type superexchange interaction and spin-orbit coupling on an
equal footing to discuss ground-state properties. We find that a zigzag ordered
state is realized in the model within nearest-neighbor interactions. We clarify
how the ordered state competes with the nonmagnetic state, which is
adiabatically connected to the quantum spin liquid state realized in a strong
spin-orbit coupling limit. Thermodynamic properties are also addressed. The
present work should provide another route to account for the Kitaev-based
magnetic properties in candidate materials.
