  We derive fundamental lower bounds on the performance of optical metrology
and communication systems in a Bayesian framework. The derivation uses
classical rate-distortion theory in conjunction with bounds on the capacity to
transmit classical information of various optical channels specified by the
system design. The bounds are expressed in terms of the system parameters, the
prior probability distribution of the parameter, and the average energy, i.e.,
number of photons $E$ in the probe state. For phase estimation, the bounds
pertain to a cyclic mean squared error criterion incorporating the cyclic
nature of the phase. In the absence of optical loss, our bounds are applicable
to multimode linear phase modulation schemes (including ancilla-assisted ones),
and to nonlinear modulations on a single mode. The bounds display
inverse-quadratic Heisenberg-limit scaling of the cyclic mean square error with
respect to $E$. In the presence of any finite amount of loss, a lower bound on
ancilla-assisted phase estimation with standard-quantum-limit (SQL) scaling is
derived, which is shown to be little different from a similar bound for
coherent-state probes. For systems involving a single optical mode, we also
obtain lower bounds on the mean squared error of estimating any classical
parameter, and on the average error probability of any $M$-ary communication
system under an average energy constraint. The bounds are valid for arbitrary
quantum measurements, for any prior probability distribution, and do not rely
on unbiasedness assumptions.
