  In this article we show that the $p$-torsion exponent of the stable stems
grows sublinearly in $n$ and the $p$-rank of the $E_2$-page of the Adams
spectral sequence grows as $\exp(\Theta( \log(n)^3))$. Together these bounds
provide the first subexponential bound on the size of the stable stems.
Conversely, we prove that a certain, precise, version of the failure of the
telescope conjecture would imply that the upper bound provided by the Adams
$E_2$-page is essentially sharp -- answering the titular question: As big as
the fate of the telescope conjecture demands.
  In an appendix joint with Andrew Senger we consider the unstable analog of
this question. Bootstrapping from the stable bounds we prove that the size of
the $p$-local homotopy groups of spheres grows like $\exp(O(\log(n)^3))$,
providing the first subexponential bound on the size of the unstable stems.
