  We study the modulational instability of small-amplitude periodic traveling
wave solutions in a generalized Ostrovsky equation. Specifically, we
investigate the invertibility of the associated linearized operator in the
vicinity of the origin and derive a modulational instability index that depends
on the dispersion and nonlinearity. We then show that the small-amplitude
periodic traveling waves of the Ostrovsky equation exhibit modulational
instability if the wavenumber is greater than a critical value which agrees
with previously found numerical results in
\cite{Whitfield2014Rotation-inducedWaves} both qualitatively and
quantitatively. We also study the effects of surface tension on modulational
instability using the index.
