  Recently, two monolayer magnetic materials, i.e., FePS3 and NiPS3, have been
successfully fabricated. Despite that they have the same atomic structure, the
two monolayers exhibit distinct magnetic properties. FePS3 holds an
out-of-plane zigzag antiferromagnetic (AFM-ZZ) structure, while NiPS3 exhibits
an in-plane AFM-ZZ structure. However, there is no theoretical model which can
properly describe its magnetic ground state due to the lack of a full
understanding of its magnetic interactions. Here, by combining the
first-principles calculations and the newly developed machine learning method,
we construct an exact spin Hamiltonian of the two magnetic materials. Different
from the previous studies which failed to fully consider the spin-orbit
coupling effect, we find that the AFM-ZZ ground state in FePS3 is stabilized by
competing ferromagnetic nearest-neighbor and antiferromagnetic third
nearest-neighbor exchange interactions, and combining single-ion anisotropy.
Whereas, the often ignored nearest-neighbor biquadratic exchange is responsible
for the in-plane AFM-ZZ ground state in NiPS3. We additionally calculate
spin-wave spectrum of AFM-ZZ structure in the two monolayers based on the exact
spin Hamiltonian, which can be directly verified by the experimental
investigation. Our work provides a theoretical framework for the origin of
AFM-ZZ ground state in two-dimensional materials.
