  We study self-oscillations of an optomechanical system, where coherent
mechanical oscillations are induced by a driven optical or microwave cavity,
for the case of an anharmonic mechanical oscillator potential. A semiclassical
analytical model is developed to characterize the limit cycle for large
mechanical amplitudes corresponding to a weak nonlinearity. As a result, we
predict conditions to achieve subpoissonian phonon statistics in the steady
state, indicating classically forbidden behavior. We compare with numerical
simulations and find very good agreement. Our model is quite general and can be
applied to other physical systems such as trapped ions or superconducting
circuits.
