  We determine the quantum ground-state properties of ultracold bosonic atoms
interacting with the mode of a high-finesse resonator. The atoms are confined
by an external optical lattice, whose period is incommensurate with the cavity
mode wave length, and are driven by a transverse laser, which is resonant with
the cavity mode. While for pointlike atoms photon scattering into the cavity is
suppressed, for sufficiently strong lasers quantum fluctuations can support the
build-up of an intracavity field, which in turn amplifies quantum fluctuations.
The dynamics is described by a Bose-Hubbard model where the coefficients due to
the cavity field depend on the atomic density at all lattice sites. Quantum
Monte Carlo simulations and mean-field calculations show that for large
parameter regions cavity backaction forces the atoms into clusters with a
checkerboard density distribution. Here, the ground state lacks superfluidity
and possesses finite compressibility, typical of a Bose-glass. This system
constitutes a novel setting where quantum fluctuations give rise to effects
usually associated with disorder.
