  We study mode competition in a multimode "phonon laser" comprised of an
optical cavity employing a highly reflective membrane as the output coupler.
Mechanical gain is provided by the intracavity radiation pressure, to which
many mechanical modes are coupled. We calculate the gain, and find that strong
oscillation in one mode suppresses the gain in other modes. For sufficiently
strong oscillation, the gain of the other modes actually switches sign and
becomes damping, a process we call "anomalous cooling." We demonstrate that
mode competition leads to single-mode operation and find excellent agreement
with our theory, including anomalous cooling.
