  The use of levitated nanospheres represents a new paradigm for the
optomechanical cooling of a small mechanical oscillator, with the prospect of
realising quantum oscillators with unprecedentedly high quality factors. We
investigate the dynamics of this system, especially in the so-called
self-trapping regimes, where one or more optical fields simultaneously trap and
cool the mechanical oscillator. The determining characteristic of this regime
is that both the mechanical frequency $\omega_M$ and single-photon
optomechanical coupling strength parameters $g$ are a function of the optical
field intensities, in contrast to usual set-ups where $\omega_M$ and $g$ are
constant for the given system. We also measure the characteristic transverse
and axial trapping frequencies of different sized silica nanospheres in a
simple optical standing wave potential, for spheres of radii $r=20-500$\,nm,
illustrating a protocol for loading single nanospheres into a standing wave
optical trap that would be formed by an optical cavity. We use this data to
confirm the dependence of the effective optomechanical coupling strength on
sphere radius for levitated nanospheres in an optical cavity and discuss the
prospects for reaching regimes of strong light-matter coupling. Theoretical
semiclassical and quantum displacement noise spectra show that for larger
nanospheres with $r \gtrsim 100$\,nm a range of interesting and novel dynamical
regimes can be accessed. These include simultaneous hybridization of the two
optical modes with the mechanical modes and parameter regimes where the system
is bistable. We show that here, in contrast to typical single-optical mode
optomechanical systems, bistabilities are independent of intracavity intensity
and can occur for very weak laser driving amplitudes.
