  In anisotropic or bianisotropic waveguides, the standard coupled mode theory
fails due to the broken link between the forward and backward propagating
modes, which together form the dual mode sets that are crucial in constructing
couple mode equations. We generalize the coupled mode theory by treating the
forward and backward propagating modes on the same footing via a generalized
eigenvalue problem that is exactly equivalent to the waveguide Hamiltonian. The
generalized eigenvalue problem is fully characterized by two operators, i.e.,
$( \bar{\bm{L}},\bar{\bm{B}})$, wherein $\bar{\bm{L}}$ is a self-adjoint
differential operator, while $\bar{\bm{B}}$ is a constant antisymmetric
operator. From the properties of $\bar{\bm{L}}$ and $\bar{\bm{B}}$, we
establish the relation between the dual mode sets that are essential in
constructing coupled mode equations in terms of forward and backward
propagating modes. By perturbation, the generalized coupled mode equation can
be derived in a natural way. Our generalized coupled mode formalism can be used
to study the mode coupling in waveguides that may contain gain/loss, anisotropy
or bianisotropy. We further illustrate how the generalized coupled theory can
be used to study the modal coupling in anisotropy and bianisotropy waveguides
through a few concrete examples.
