  We have observed nonlinear transduction of the thermomechanical motion of a
nanomechanical resonator when detected as laser transmission through a sideband
unresolved optomechanical cavity. Nonlinear detection mechanisms are of
considerable interest as special cases allow for quantum nondemolition
measurements of the mechanical resonator's energy. We investigate the origin of
the nonlinearity in the optomechanical detection apparatus and derive a
theoretical framework for the nonlinear signal transduction, and the optical
spring effect, from both nonlinearities in the optical transfer function and
second order optomechanical coupling. By measuring the dependence of the linear
and nonlinear signal transduction -- as well as the mechanical frequency shift
-- on laser detuning from optical resonance, we provide estimates of the
contributions from the linear and quadratic optomechanical couplings.
