  A scheme is presented to optimize the optomechanical cooling of mechanical
resonator in instability regime. Based on the stability analysis, we uncovered
a distinct bistable effect of photons and phonons, which can be used to realize
a strong nonlinear effect even in the single-photon weak coupling regime.
Considering the experimental realization, we investigate the sideband cooling
in bistable regime with and without quantum nonlinearity. It is shown that the
fluctuation of the steady state phonons can be excellently suppressed at a
rather low level due to the anti-rotating-wave effect, and it does not require
high quality factor of the cavity. Our scheme offers a new perspective for
optimizing the sideband cooling of mechanical resonators in the weak coupling
regime.
