  We perform 3D radiation hydrodynamic local shearing box simulations to study
the outcome of gravitational instability (GI) in optically thick Active
Galactic Nuclei (AGN) accretion disks. GI develops when the Toomre parameter QT
\leq 1, and may lead to turbulent heating that balances radiative cooling.
However, when radiative cooling is too efficient, the disk may undergo runaway
gravitational fragmentation. In the fully gas-pressure-dominated case, we
confirm the classical result that such a thermal balance holds when the
Shakura-Sunyaev viscosity parameter (alpha) due to the gravitationally-driven
turbulence is \sim 0.2, corresponding to dimensionless cooling times Omega
tcool \sim 5. As the fraction of support by radiation pressure increases, the
disk becomes more prone to fragmentation, with a reduced (increased) critical
value of alpha (omega tcool). The effect is already significant when the
radiation pressure exceeds 10% of the gas pressure, while fully
radiation-pressure-dominated disks fragment at Omega tcool <50 . The latter
translates to a maximum turbulence level alpha<0.02, comparable to that
generated by Magnetorotational Instability (MRI). Our results suggest that
gravitationally unstable (QT \sim 1) outer regions of AGN disks with
significant radiation pressure (likely for high/near- Eddington accretion
rates) should always fragment into stars, and perhaps black holes.
