  In this paper, we study theoretically a hybrid optomechanical system
consisting of a degenerate optical parametric amplifier inside a driven optical
cavity with a moving end mirror which is modeled as a stiffening Duffing-like
anharmonic quantum mechanical oscillator. By providing analytical expressions
for the critical values of the system parameters corresponding to the emergence
of the multistability behavior in the steady-state response of the system, we
show that the stiffening mechanical Duffing anharmonicity reduces the width of
the multistability region while the optical parametric nonlinearity can be
exploited to drive the system toward the multistability region. We also show
that for appropriate values of the mechanical anharmonicity strength the
steady-state mechanical squeezing and the ground-state cooling of the
mechanical resonator can be achieved. Moreover, we find that the presence of
the nonlinear gain medium can lead to the improvement of the mechanical
anharmonicity-induced cooling of the mechanical motion, as well as to the
mechanical squeezing beyond the standard quantum limit of 3 dB.
