  We develop a quasilinear theory of the Vlasov equation in order to describe
the approach of systems with long-range interactions to quasi-stationary
states. We derive a diffusion equation governing the evolution of the velocity
distribution of the system towards a steady state. This steady state is
expected to correspond to the angle-averaged quasi-stationary distribution
function reached by the Vlasov equation as a result of a violent relaxation. We
compare the prediction of the quasilinear theory to direct numerical
simulations of the Hamiltonian Mean Field model, starting from an unstable
spatially homogeneous distribution, either Gaussian or semi-elliptical. We find
that the quasilinear theory works reasonably well for weakly unstable initial
conditions and that it is able to predict the energy marking the
out-of-equilibrium phase transition between unmagnetized and magnetized
quasi-stationary states. At energies lower than the out-of-equilibrium
transition the quasilinear theory works less well, the disagreement with the
numerical simulations increasing by decreasing the energy. In that case, we
observe, in agreement with our previous numerical study [A. Campa and P.-H.
Chavanis, Eur. Phys. J. B 86, 170 (2013)], that the quasi-stationary states are
remarkably well fitted by polytropic distributions (Tsallis distributions) with
index $n=2$ (Gaussian case) or $n=1$ (semi-elliptical case). In particular,
these polytropic distributions are able to account for the region of negative
specific heats in the out-of-equilibrium caloric curve, unlike the Boltzmann
and Lynden-Bell distributions.
