  In this paper we reveal the rich ground-state properties induced by the
strong nonlinear atom-photon interaction, which has been found in the recent
experiment about a Bose-Einstein condensate coupling with a high-finesse cavity
[Nature (London) 464, 1301 (2010)]. Two detuning-dependent phase diagrams are
revealed by investigating the experimentally-measurable atomic population. In
particular, two new quantum phase transitions from the superradiant phase or
the normal phase to a dynamically-unstable phase are predicted in the blue
detuning, and moreover, the three-phase coexistence points are found. It is
also demonstrated that these predicted quantum phase transitions are the
intrinsic transitions governed only by the second-order derivative of the
ground-state energy. Finally, we also point out that the region involving
coexistence of the superradiant phase and the normal phase predicted before can
not happen in the ground state.
