  Antiferromagnetic quantum spin systems can exhibit a transition between
collinear and spiral ground states, driven by frustration. Classically this is
a smooth crossover and the crossover point is termed a Lifshitz point. Quantum
fluctuations change the nature of the transition. In particular it has been
argued previously that in the two-dimensional (2D) case a spin liquid (SL)
state is developed in the vicinity of the Lifshitz point, termed a Lifshitz SL.
In the present work, using a field theory approach, we solve the Lifshitz
quantum phase transition problem for the 2D frustrated XY-model. Specifically,
we show that, unlike the SU(2) symmetric Lifshitz case, in the XY-model the SL
exists only at the critical point. At zero temperature we calculate
nonuniversal critical exponents in the Neel and in the spin spiral state and
relate these to properties of the SL. We also solve the transition problem at a
finite temperature and discuss the role of topological excitations.
