  We consider a generic optomechanical system, consisting of a driven optical
cavity and a movable mirror attached to a cantilever. Systems of this kind (and
analogues) have been realized in many recent experiments. It is well known that
those systems can exhibit an instability towards a regime where the cantilever
settles into self-sustained oscillations. In this paper, we briefly review the
classical theory of the optomechanical instability, and then discuss the
features arising in the quantum regime. We solve numerically a full quantum
master equation for the coupled system, and use it to analyze the photon
number, the cantilever's mechanical energy, the phonon probability distribution
and the mechanical Wigner density, as a function of experimentally accessible
control parameters. We observe and discuss the quantum-to-classical transition
as a function of a suitable dimensionless quantum parameter.
