  Feedback is a powerful and ubiquitous technique both in classical and quantum
system control. Its standard implementation relies on measuring the state of a
system, processing the classical signal, and feeding it back to the system. In
quantum physics, however, measurements not only read out the state of the
system but also modify it irreversibly. Coherent feedback is a different kind
of feedback that coherently processes and feeds back quantum signals without
actually measuring the system. Here, we report on the experimental realization
and the theoretical analysis of an optical coherent feedback platform to
control the motional state of a nanomechanical membrane in an optical cavity.
The coherent feedback loop consists of a light field interacting twice with the
same mechanical mode through different cavity modes, without {performing any}
measurement. Tuning the optical phase and delay of the feedback loop allows us
to control the motional state of the mechanical oscillator, its resonance
frequency and also its damping rate, which we use to cool the membrane close to
the quantum ground state. Our theoretical analysis provides the optimal cooling
conditions, showing that this new technique enables ground-state cooling.
Experimentally, we show that we can cool the membrane to a state with
$\bar{n}_m = 4.89 \pm 0.14 $ phonons (${480}\,{\mu \mathrm{K}}$) in a
${20}\,\mathrm{K}$ environment. This lies below the theoretical limit of cavity
dynamical backaction cooling in the unresolved sideband regime and is achieved
with only 1$\%$ of the optical power required for cavity cooling. Our feedback
scheme is very versatile, offering new opportunities for quantum control in a
variety of optomechanical systems.
