  The transverse-field Ising model on the triangular lattice is expected to
host an intermediate finite-temperature Kosterlitz-Thouless (KT) phase through
a mapping of the spins on each triangular unit to a complex order parameter.
TmMgGaO$_4$ is a candidate material to realize such physics due to the
non-Kramers nature of Tm$^{3+}$ ion and the resulting two-singlet single-ion
ground state. Using inelastic neutron scattering, we confirm this picture by
determining the leading parameters of the low-energy effective Hamiltonian of
TmMgGaO$_4$. Subsequently, we track the predicted KT phase and related
transitions by inspecting the field and temperature dependence of the ac
susceptibility. We further probe the spin correlations in both reciprocal space
and real space via single crystal neutron diffraction and magnetic total
scattering techniques, respectively. Magnetic pair distribution function
analysis provides evidence for the formation of vortex-antivortex pairs that
characterize the proposed KT phase around 5~K. Although structural disorder
influences the field-induced behavior of TmMgGaO$_4$, the magnetism in zero
field appears relatively free from these effects. These results position
TmMgGaO$_4$ as a strong candidate for a solid-state realization of KT physics
in a dense spin system.
