  We prove that the modulational instability criterion of the formal Whitham
modulation theory agrees with the spectral stability of long wavelength
perturbations of periodic travelling wave solutions to the generalized Whitham
equation. We use the standard WKB procedure to derive a quasi-linear system of
three Whitham modulation equations, written in terms of the mass, momentum, and
wave number of a periodic travelling wave solution. We use the same quantities
as parameters in a rigorous spectral perturbation of the linearized operator,
which allows us to track the bifurcation of the zero eigenvalue as the Floquet
parameter varies. We show that the hyperbolicity of the Whitham system is a
necessary condition for the existence of purely imaginary eigenvalues in the
linearized system, and hence also a prerequisite for modulational stability of
the underlying wave. Since the generalized Whitham equation has a Hamiltonian
structure, we conclude that strict hyperbolicity is a sufficient condition for
modulational stability.
