  Using the Polyakov-Nambu-Jona-Lasinia (PNJL) model, we study the spinodal
instability of a baryon-rich quark-gluon plasma in the linear response theory.
We find that the spinodal unstable region in the temperature and density plane
shrinks with increasing wave number of the unstable mode and is also reduced if
the effect of Polyakov loop is not included. In the small wave number or long
wavelength limit, the spinodal boundaries in both cases of with and without the
Polyakov loop coincide with those determined from the isothermal spinodal
instability in the thermodynamic approach. Also, the vector interactions among
quarks is found to suppress unstable modes of all wave numbers. Moreover, the
growth rate of unstable modes initially increases with the wave number but is
reduced when the wave number becomes large. Including the collisional effect
from quark scattering via the linearized Boltzmann equation, we further find
that it decreases the growth rate of unstable modes of all wave numbers.
Relevance of these results to relativistic heavy ion collisions is discussed.
