  Three-mode parametric interactions occur in triply-resonant optomechanical
systems: photons from an optical pump mode are coherently scattered to a
high-order mode by mechanical motion of the cavity mirrors, and these modes
resonantly interact via radiation pressure force when certain conditions are
met. Such effects are predicted to occur in long baseline advanced
gravitational-wave detectors. They can pump energy into acoustic modes, leading
to parametric instability, but they can also extract acoustic energy, leading
to optomechanical cooling. We develop a large amplitude model of three-mode
interactions that explains the ring-up amplitude saturation after instability
occurs. We also demonstrate both radiation-pressure cooling and mechanical
amplification in two different three-mode optomechanical systems, including the
first observation of the three-mode parametric instability in a free-space
Fabry-Perot cavity. The experimental data agrees well with the theoretical
model. Contrary to expectations, parametric instability does not lead to loss
of cavity lock, a fact which may make it easier to implement control techniques
to overcome instability.
