  We investigate the nonlinear light-matter interaction of a Bose-Einstein
condensate trapped in an external periodic potential inside an optical cavity
which is weakly coupled to vacuum radiation modes and driven by a transverse
pump field. Based on a generalized Bose-Hubbard model which incorporates a
single cavity mode, we include the collective backaction of the atoms on the
cavity light field and determine the nonequilibrium quantum phases within the
nonperturbative bosonic dynamical mean-field theory.With the system parameters
adapted to recent experiments, we find a quantum phase transition from a normal
phase to a self-organized superfluid phase, which is related to the
Hepp-Lieb-Dicke superradiance phase transition. For even stronger pumping, a
self-organized Mott insulator phase arises.
