  We study the stress tensor four-point function for $\mathcal{N}=4$ SYM with
gauge group $G=SU(N)$, $SO(2N+1)$, $SO(2N)$ or $USp(2N)$ at large $N$. When
$G=SU(N)$, the theory is dual to type IIB string theory on $AdS_5\times S^5$
with complexified string coupling $\tau_s$, while for the other cases it is
dual to the orbifold theory on $AdS_5\times S^5/\mathbb{Z}_2$. In all cases we
use the analytic bootstrap and constraints from localization to compute 1-loop
and higher derivative tree level corrections to the leading supergravity
approximation of the correlator. We give perturbative evidence that the
localization constraint in the large $N$ and finite complexified coupling
$\tau$ limit can be written for each $G$ in terms of Eisenstein series that are
modular invariant in terms of $\tau_s\propto\tau$, which allows us to fix
protected terms in the correlator in that limit. In all cases, we find that the
flat space limit of the correlator precisely matches the type IIB S-matrix. We
also find a closed form expression for the $SU(N)$ 1-loop Mellin amplitude with
supergravity vertices. Finally, we compare our analytic predictions at large
$N$ and finite $\tau$ to bounds from the numerical bootstrap in the large $N$
regime, and find that they are not saturated for any $G$ and any $\tau$, which
suggests that no physical theory saturates these bootstrap bounds.
