  Optomechanical systems can exhibit self-sustained limit cycles where the
quantum state of the mechanical resonator possesses nonclassical
characteristics such as a strongly negative Wigner density, as was shown
recently in a numerical study by Qian et al. [Physical Review Letters, 109,
253601 (2012)]. Here we derive a Fokker-Planck equation describing mechanical
limit cycles in the quantum regime which correctly reproduces the numerically
observed nonclassical features. The derivation starts from the standard
optomechanical master equation, and is based on techniques borrowed from the
laser theory due to Haake's and Lewenstein. We compare our analytical model
with numerical solutions of the master equation based on Monte-Carlo
simulations, and find very good agreement over a wide and so far unexplored
regime of system parameters. As one main conclusion, we predict negative Wigner
functions to be observable even for surprisingly classical parameters, i.e.
outside the single-photon strong coupling regime, for strong cavity drive, and
rather large limit cycle amplitudes. The general approach taken here provides a
natural starting point for further studies of quantum effects in optomechanics.
