  Interlayer magnetic interactions play a pivotal role in determining the
magnetic arrangement within van der Waals (vdW) magnets, and the remarkable
tunability of these interactions through applied pressure further enhances
their significance. Here, we investigate NiI2 flakes, a representative vdW
magnet, under hydrostatic pressures up to 11 GPa. We reveal a notable increase
in magnetic transition temperatures for both helimagnetic and antiferromagnetic
states, and find that a reversible transition from helimagnetic to
antiferromagnetic (AFM) phases at approximately 7 GPa challenges established
theoretical and experimental expectations. While the increase in transition
temperature aligns with pressure-enhanced overall exchange interaction
strengths, we identify the significant role of the second-nearest neighbor
interlayer interaction, which competes with intra-layer frustration and favors
the AFM state as demonstrated in the Monte Carlo simulations. Experimental and
simulated results converge on the existence of an intermediate helimagnetic
ordered state in NiI2 before transitioning to the AFM state. These findings
underscore the pivotal role of interlayer interactions in shaping the magnetic
ground state, providing fresh perspectives for innovative applications in
nanoscale magnetic device design.
