  We will derive the general theory of the beam-breakup instability in
recirculating linear accelerators with coupled beam optics and with polarized
higher order dipole modes. The bunches do not have to be at the same RF phase
during each recirculation turn. This is important for the description of energy
recovery linacs (ERLs) where beam currents become very large and coupled optics
are used on purpose to increase the threshold current. A remarkable agreement
with tracking data is demonstrated. The general formulas are then analyzed for
several analytically solvable cases, which show: (a) Why different higher order
modes (HOM) in one cavity can couple and cannot be considered individually,
even when their frequencies are separated by much more than the resonance
widths of the HOMs. For the Cornell ERL as an example, it is noted that optimum
advantage of coupled optics is taken when the cavities are designed with an
$x$-$y$ HOM frequency splitting of above 50MHz. The threshold current is then
far above the design current of this accelerator. (b) How the $x$-$y$ coupling
in the particle optics determines when modes can be considered separately. (c)
That the increase of the threshold current obtainable by coupled optics and
polarized modes diminishes roughly with the square root of the HOMs' quality
factors. Therefore the largest advantages are achieved with cavities that are
not specifically designed to minimize these quality factors, e.g. by means of
HOM absorbers. (d) How multiple-turn recirculation interferes with the
threshold improvements obtained with a coupled optics.
