  We develop a systematic field theoretic description for the roughness
correction to the Casimir free energy of parallel plates. Roughness is modeled
by specifying a generating functional for correlation functions of the height
profile, the two-point correlation function being characterized by the
variance, \sigma^2, and correlation length, \ell, of the profile. We obtain the
partition function of a massless scalar quantum field interacting with the
height profile of the surface via a \delta-function potential. The partition
function of this model is also given by a holographic reduction to three
coupled scalar fields on a two-dimensional plane. The original
three-dimensional space with a parallel plate at separation 'a' is encoded in
the non-local propagators of the surface fields on its boundary. Feynman rules
for this equivalent 2+1-dimensional model are derived and its counter terms
constructed. The two-loop contribution to the free energy of this model gives
the leading roughness correction. The absolute separation to a rough plate is
measured to an effective plane that is displaced a distance \rho\propto
\sigma^2/\ell from the mean of its profile. This definition of the separation
eliminates corrections to the free energy of order 1/a^4 and results in a
unitary model. We derive an effective low-energy theory in the limit \ell\ll a.
It gives the scattering matrix and effective planar surface of a very rough
plate in terms of the single length scale \rho. The Casimir force on a rough
plate is found to always weaken with decreasing correlation length \ell. The
two-loop approximation to the free energy interpolates between the free energy
of the effective low-energy model and that obtained in proximity force
approximation -- the force on a very rough plate with \sigma\gtrsim 0.5\ell
being weaker than on a flat plate at any separation.
