  We calculate the second order roughness correction to the Casimir energy for
two parallel metallic mirrors. Our results may also be applied to the
plane-sphere geometry used in most experiments. The metallic mirrors are
described by the plasma model, with arbitrary values for the plasma wavelength,
the mirror separation and the roughness correlation length, with the roughness
amplitude remaining the smallest length scale for perturbation theory to hold.
  From the analysis of the intracavity field fluctuations, we obtain the
Casimir energy correction in terms of generalized reflection operators, which
account for diffraction and polarization coupling in the scattering by the
rough surfaces. We present simple analytical expressions for several limiting
cases, as well as numerical results that allow for a reliable calculation of
the roughness correction in real experiments. The correction is larger than the
result of the Proximity Force Approximation, which is obtained from our theory
as a limiting case (very smooth surfaces).
