  We investigate the near-critical dynamics of atomic density fluctuations in
the non-equilibrium self-organization transition of an optically driven quantum
gas coupled to a single mode of a cavity. In this system cavity-mediated
long-range interactions between atoms, tunable by the drive strength, lead to
softening of an excitation mode recently observed in experiments. This
phenomenon has previously been studied within a two-mode approximation for the
collective motional degrees of freedom of the atomic condensate which results
in an effective open-system Dicke model. Here, including the full spectrum of
atomic modes we find a finite lifetime for a roton-like mode in the Bogoliubov
excitation spectrum that is strongly pump-dependent. The corresponding decay
rate and critical exponents for the phase-transition are calculated explaining
the non-monotonic pump-dependent atomic damping rate observed in recent
experiments. We compute the near-critical behavior of the intra-cavity field
fluctuations, that has been previously shown to be enhanced with respect to the
equilibrium Dicke model in a two-mode approximation. We highlight the role of
the finite size of the system in the suppression of it below the expectations
of the open Dicke model.
