  We analyze the QCD dynamics of diffractive deep inelastic scattering. The
presence of a rapidity gap between the target and diffractive system requires
that the target remnant emerges in a color singlet state, which we show is made
possible by the soft rescattering of the struck quark. This rescattering is
described by the path-ordered exponential (Wilson line) in the expression for
the parton distribution function of the target. The multiple scattering of the
struck parton via instantaneous interactions in the target generates dominantly
imaginary diffractive amplitudes, giving rise to an "effective pomeron"
exchange. The pomeron is not an intrinsic part of the proton but a dynamical
effect of the interaction. This picture also applies to diffraction in
hadron-initiated processes. Due to the different color environment the
rescattering is different in virtual photon- and hadron-induced processes,
explaining the observed non-universality of diffractive parton distributions.
This framework provides a theoretical basis for the phenomenologically
successful Soft Color Interaction model which includes rescattering effects and
thus generates a variety of final states with rapidity gaps. We discuss
developments of the SCI model to account for the color coherence features of
the underlying subprocesses.
