  We explore the nonlinear dynamics of a cavity optomechanical system. Our
realization consisting of a drumhead nano-electro-mechanical resonator (NEMS)
coupled to a microwave cavity, allows for a nearly ideal platform to study the
nonlinearities arising purely due to radiation-pressure physics. Experiments
are performed under a strong microwave Stokes pumping which triggers mechanical
self-sustained oscillations. We analyze the results in the framework of an
extended nonlinear optomechanical theory, and demonstrate that quadratic and
cubic coupling terms in the opto-mechanical Hamiltonian have to be considered.
Quantitative agreement with the measurements is obtained considering only
genuine geometrical nonlinearities: no thermo-optical instabilities are
observed, in contrast with laser-driven systems. Based on these results, we
describe a method to quantify nonlinear properties of microwave optomechanical
devices. Such a technique, available now in the quantum electro-mechanics
toolbox, but completely generic, is mandatory for the development of new
schemes where higher-order coupling terms are proposed as a new resource, like
Quantum Non-Demolition measurements, or in the search for new fundamental
quantum signatures, like Quantum Gravity. We also find that the motion imprints
a wide comb of extremely narrow peaks in the microwave output field, which
could also be exploited in specific microwave-based measurements, potentially
limited only by the quantum noise of the optical and the mechanical fields for
a ground-state cooled NEMS device.
