  Enhanced nonlinear optical response of a coherent atomic medium is the basis
for many atomic sensors, and their performance is ultimately limited by the
quantum fluctuations of the optical read-out. Here we demonstrate that
off-resonant interactions can significantly modify the quantum noise of the
optical field, even when their effect on the mean signal is negligible. We
illustrate this concept by using an atomic magnetometer based on the nonlinear
Faraday effect: the rotation of the light polarization is mainly determined by
the resonant light-induced spin alignment, which alone does not change the
photon statistics of the optical probe. Yet, we found that the minimum noise of
output polarization rotation measurements is above the expected shot noise
limit. This excess quantum noise is due to off-resonant coupling and grows with
atomic density. We also show that the detection scheme can be modified to
reduce the measured quantum noise (even below the shot-noise limit) but only at
the expense of the reduced rotational sensitivity. These results show the
existence of previously unnoticed factors in fundamental limitations in atomic
magnetometry and could have impacts in many other atom-light based precision
measurements.
