  Optomechanics concerns with the coupling between optical cavities and
mechanical resonators. Most early works are concentrated in the physics of
optomechanics in the small-displacement regime and consider one single optical
cavity mode participating in the optomechanical coupling. In this paper, we
focus on optomechanics in the extremely-large-amplitude regime in which a
mechanical resonator is coupled with multiple optical cavity modes during the
oscillation. We explicitly show that the mechanical resonator can present
self-sustained oscillations in a novel way with limit cycles in the shape of
sawtooth-edged ellipses and exhibit dynamical multistability. By analyzing the
mechanical oscillation process and the accompanied variation of the optical
cavity occupation, we develop an energy-balanced condition to ensure the
stability of self-sustained oscillation. The effect of the mechanical
nonlinearities on the dynamics of the mechanical resonator is also
investigated.
