  Motivated by recent experiments on the frustrated quantum magnetic compound
YbMgGaO4, we study an effective spin model on triangular lattice taking into
account the effects of the spin-orbit coupling. We determine the classical
ground-state phase diagram of this model, which includes a 120 degree Neel and
two collinear antiferromagnetic phases. In the vicinity of the phase boundary
between the Neel and collinear phases, we identify three intermediate
non-collinear antiferromagnetic phases. In each of them the magnetic moments
are ordered at multiple incommensurate wave vector Q values. We further study
the effects of quantum fluctuations in this model via a linear spin-wave
theory. We find that the spin excitation gap of the non-collinear multi-Q
antiferromagnetic state is finite but can be vanishingly small, and this state
is unstable to a spin liquid phase under strong quantum fluctuations in some
large J_{z+-} regime.
