  We study modulational stability and instability in the Whitham equation,
combining the dispersion relation of water waves and a nonlinearity of the
shallow water equations, and modified to permit the effects of surface tension
and constant vorticity. When the surface tension coefficient is large, we show
that a periodic traveling wave of sufficiently small amplitude is unstable to
long wavelength perturbations if the wave number is greater than a critical
value, and stable otherwise, similarly to the Benjamin-Feir instability of
gravity waves. In the case of weak surface tension, we find intervals of stable
and unstable wave numbers, whose boundaries are associated with the extremum of
the group velocity, the resonance between the first and second harmonics, the
resonance between long and short waves, and a resonance between dispersion and
the nonlinearity. For each constant vorticity we show that a periodic traveling
wave of sufficiently small amplitude is unstable if the wave number is greater
than a critical value, and stable otherwise. Moreover it can be made stable for
a sufficiently large vorticity. The results agree with those based upon
numerical computations or formal multiple-scale expansions for the physical
problem.
