  We study and compare the critical properties of the two-dimensional (2D) XY
model in a transverse magnetic field with magnetic filling factors f=1/3 and
f=2/5. In addition to the spin waves, the low energy excitations of the system
consist of various domain walls between degenerate ground states. The lowest
energy domain wall has a similar structure for both f=1/3 and f=2/5 and its
properties dictate the nature of the phase transition. For f=2/5 these lowest
energy walls have a negative energy for binding to each other, giving rise to a
branching domain-wall structure and leading to a first order phase transition.
For f=1/3 this binding energy is positive, resulting in a linear critical
interface. In order to make a comparison to recent experiments, we investigate
the effect of small quenched bond disorder for f=2/5. A finite-size scaling
analysis of extensive Monte Carlo simulations strongly suggests that the
critical exponents of the phase transition for f=1/3, and for f=2/5 with
disorder, fall into the universality class of the two-dimensional Ising model.
