  The inherently nonlinear interaction between light and motion in cavity
optomechanical systems has experimentally been studied in a linearized
description in all except highly driven cases. Here we demonstrate a nanoscale
optomechanical system, in which the interaction between light and motion is so
large (single-photon cooperativity $C_0 \approx 10^3$) that thermal motion
induces optical frequency fluctuations larger than the intrinsic optical
linewidth. The system thereby operates in a fully nonlinear regime, which
pronouncedly impacts the optical response, displacement measurement, and
radiation pressure backaction. Experiments show that the apparent optical
linewidth is dominated by thermomechanically-induced frequency fluctuations
over a wide temperature range. The nonlinearity induces breakdown of the
traditional cavity optomechanical descriptions of thermal displacement
measurements. Moreover, we explore how radiation pressure backaction in this
regime affects the mechanical fluctuation spectra. The strong nonlinearity
could serve as a resource to control the motional state of the resonator. We
demonstrate the use of highly nonlinear transduction to perform a quadratic
measurement of position while suppressing linear transduction.
