  Sr2IrO4 is a spin-orbit coupled insulator with an antiferromagnetic (AFM)
transition at TN=240 K. We report results of a comprehensive study of
single-crystal Sr2Ir1-xTbxO4. This study found that mere 3% (x=0.03)
tetravalent Tb4+(4f7) substituting for Ir4+ (rather than Sr2+) completely
suppresses the long-range collinear AFM transition but retains the insulating
state, leading to a phase diagram featuring a decoupling of magnetic
interactions and charge gap. The insulating state at x=0.03 is characterized by
an unusually large specific heat at low temperatures and an incommensurate
magnetic state having magnetic peaks at (0.95, 0, 0) and (0, 0.95, 0) in the
neutron diffraction, suggesting a spiral or spin density wave order. It is
apparent that Tb doping effectively changes the relative strength of the SOI
and the tetragonal CEF and enhances the Hund's rule coupling that competes with
the SOI, and destabilizes the AFM state. However, the disappearance of the AFM
accompanies no metallic state chiefly because an energy level mismatch for the
Ir and Tb sites weakens charge carrier hopping and renders a persistent
insulating state. This work highlights an unconventional correlation between
the AFM and insulating states in which the magnetic transition plays no
critical role in the formation of the charge gap in the iridate.
