  We investigate a paradigm example of cavity quantum electrodynamics with many
body systems: an ultracold atomic gas inside a pumped optical resonator. In
particular, we study the stability of atomic insulator-like states, confined by
the mechanical potential emerging from the cavity field spatial mode structure.
As in open space, when the optical potential is sufficiently deep, the atomic
gas is in the Mott-like state. Inside the cavity, however, the potential
depends on the atomic distribution, which determines the refractive index of
the medium, thus altering the intracavity field amplitude. We derive the
effective Bose-Hubbard model describing the physics of the system in one
dimension and study the crossover between the superfluid -- Mott insulator
quantum states. We determine the regions of parameters where the atomic
insulator states are stable, and predict the existence of overlapping stability
regions corresponding to competing insulator-like states. Bistable behavior,
controlled by the pump intensity, is encountered in the vicinity of the shifted
cavity resonance.
