  Since its elaboration by Whitham, almost fifty years ago, modulation theory
has been known to be closely related to the stability of periodic traveling
waves. However, it is only recently that this relationship has been elucidated,
and that fully nonlinear results have been obtained. These only concern
dissipative systems though: reaction-diffusion systems were first considered by
Doelman, Sandstede, Scheel, and Schneider [Mem. Amer. Math. Soc. 2009], and
viscous systems of conservation laws have been addressed by Johnson, Noble,
Rodrigues, and Zumbrun [preprint 2012]. Here, only nondissipative models are
considered, and a most basic question is investigated, namely the expected link
between the hyperbolicity of modulated equations and the spectral stability of
periodic traveling waves to sideband perturbations. This is done first in an
abstract Hamiltonian framework, which encompasses a number of dispersive
models, in particular the well-known (generalized) Korteweg--de Vries equation,
and the less known Euler--Korteweg system, in both Eulerian coordinates and
Lagrangian coordinates. The latter is itself an abstract framework for several
models arising in water waves theory, superfluidity, and quantum hydrodynamics.
As regards its application to compressible capillary fluids, attention is paid
here to untangle the interplay between traveling waves/modulation equations in
Eulerian coordinates and those in Lagrangian coordinates. In the most general
setting, it is proved that the hyperbolicity of modulated equations is indeed
necessary for the spectral stability of periodic traveling waves. This extends
earlier results by Serre [Comm. Partial Differential Equations 2005], Oh and
Zumbrun [Arch. Ration. Mech. Anal. 2003], and Johnson, Zumbrun and Bronski
[Phys. D 2010]. In addition, reduced necessary conditions are obtained in the
small amplitude limit. Then numerical investigations are carried out for the
modulated equations of the Euler--Korteweg system with two types of 'pressure'
laws, namely the quadratic law of shallow water equations, and the nonmonotone
van der Waals pressure law. Both the evolutionarity and the hyperbolicity of
the modulated equations are tested, and regions of modulational instability are
thus exhibited.
