  With an increasing coupling between light and mechanics, nonlinearities begin
to play an important role in optomechanics. We solve the quantum dynamics of an
optomechanical system in the multi-photon strong coupling regime retaining
nonlinear terms. This is achieved by performing a Schrieffer-Wolff
transformation on the Hamiltonian including driving terms. The approach is
valid away from the red- and blue-sideband drive. We show that the mechanical
resonator displays self-sustained oscillations in regimes where the linearized
model predicts instabilities, and that the amplitude of these oscillations is
limited by the nonlinear terms. Related oscillations of the photon number are
present due to frequency mixing of the shifted mechanical and cavity
frequencies. This leads to additional peaks in the cavity's power spectral
density. Furthermore, we show that it is possible to create phonon states with
sub-Poissonian statistics when the system is red-detuned. This result is valid
even with strong driving and with initial coherent states.
