  Breaking the time-reversal symmetry of light is of great importance for
fundamental physics and has attracted increasing interest in the study of
non-reciprocal photonic devices. Here, we experimentally demonstrate a chiral
cavity QED system with multiple atoms strongly coupled to a Fabry-Perot cavity.
By polarizing the internal quantum state of the atoms, the time-reversal
symmetry of the atom-cavity interaction is broken. The strongly coupled
atom-cavity system can be described by non-reciprocal quasiparticles, i.e., the
cavity polariton. When it works in the linear regime, the inherent
nonreciprocity makes the system work as a single-photon-level optical isolator.
Benefiting from the collective enhancement of multiple atoms, an isolation
ratio exceeding 30~dB on the single-quanta level (~0.1 photon on average) is
achieved. The validity of the non-reciprocal device under zero magnetic field
and the reconfigurability of the isolation direction are also experimentally
demonstrated. Moreover, when the cavity polariton works in the nonlinear
regime, the quantum interference between polaritons with weak anharmonicity
induces non-reciprocal nonclassical statistics of cavity transmission from
coherent probe light.
