  The rare-earth material $\mathrm{LiHoF_4}$ is believed to be an experimental
realization of the celebrated (dipolar) Ising model, and upon the inclusion of
a transverse field $B_x$, an archetypal quantum Ising model. Moreover, by
substituting the magnetic Ho ions by non-magnetic Y ions, disorder can be
introduced into the system giving rise to a dipolar disordered magnet and at
high disorders to a spin-glass. Indeed, this material has been scrutinized
experimentally, numerically and theoretically over many decades with the aim of
understanding various collective magnetic phenomena. One of the to-date open
questions is the discrepancy between the experimental and theoretical $B_x -T$
phase diagram at low-fields and high temperatures. Here we propose a mechanism,
backed by numerical results, that highlights the importance of quantum
fluctuations induced by the off-diagonal dipolar terms, in determining the
critical temperature of anisotropic dipolar magnets in the presence and in the
absence of a transverse field. We thus show that the description as a simple
Ising system is insufficient to quantitatively describe the full phase diagram
of $\mathrm{LiHoF_4}$, for the pure as well as for the dilute system.
