  How a certain ground state of complex physical systems emerges, especially in
two-dimensional materials, is a fundamental question in condensed-matter
physics. A particularly interesting case is systems belonging to the class of
XY Hamiltonian where the magnetic order parameter of conventional nature is
unstable in two-dimensional materials leading to a
Berezinskii-Kosterlitz-Thouless transition. Here, we report how the XXZ-type
antiferromagnetic order of a magnetic van der Waals material, NiPS3, behaves
upon reducing the thickness and ultimately becomes unstable in the monolayer
limit. Our experimental data are consistent with the findings based on
renormalization group theory that at low temperatures a two-dimensional XXZ
system behaves like a two-dimensional XY one, which cannot have a long-range
order at finite temperatures. This work provides experimental examination of
the XY magnetism in the atomically thin limit and opens new opportunities of
exploiting these fundamental theorems of magnetism using magnetic van der Waals
materials.
