  Advanced gravitational wave detectors use suspended test masses to form
optical resonant cavities for enhancing the detector sensitivity. These
cavities store hundreds of kilowatts of coherent light and even higher optical
power for future detectors. With such high optical power, the radiation
pressure effect inside the cavity creates sufficiently strong coupling between
test masses whose dynamics are significantly altered. The dynamics of two
independent nearly free masses become a coupled mechanical resonator system.
The transfer function of the local control system used for controlling the test
masses is modified by the radiation pressure effect. The changes in the
transfer function of the local control systems can result in a new type of
angular instability which occurs at only 1.3 \% of the Sidles-Sigg instability
threshold power. We report experimental results on a 74~m suspended cavity with
a few kilowatts of circulating power, for which the power to mass ratio is
comparable to the current Advanced LIGO. The radiation pressure effect on the
test masses behaves like an additional optical feedback with respect to the
local angular control, potentially making the mirror control system unstable.
When the local angular control system is optimized for maximum stability
margin, the instability threshold power increases from 4~kW to 29~kW. The
system behavior is consistent with our simulation and the power dependent
evolution of both the cavity soft and hard mode is observed. We show that this
phenomenon is likely to significantly affect proposed gravitational wave
detectors that require very high optical power.
