  Three mode parametric instability has been predicted in Advanced
gravitational wave detectors. Here we present the first observation of this
phenomenon in a large scale suspended optical cavity designed to be comparable
to those of advanced gravitational wave detectors. Our results show that
previous modelling assumptions that transverse optical modes are stable in
frequency except for frequency drifts on a thermal deformation time scale is
unlikely to be valid for suspended mass optical cavities. We demonstrate that
mirror figure errors cause a dependence of transverse mode offset frequency on
spot position. Combined with low frequency residual motion of suspended
mirrors, this leads to transverse mode frequency modulation which suppresses
the effective parametric gain. We show that this gain suppression mechanism can
be enhanced by laser spot dithering or fast thermal modulation. Using Advanced
LIGO test mass data and thermal modelling we show that gain suppression factors
of 10-20 could be achieved for individual modes, sufficient to greatly
ameliorate the parametric instability problem.
