  Quantum speed limits such as the Mandelstam-Tamm or Margolus-Levitin bounds
offer a quantitative formulation of the energy-time uncertainty principle that
constrains dynamics over short times. We show that the spectral form factor, a
central quantity in quantum chaos, sets a universal state-independent bound on
the quantum dynamics of a complete set of initial states over arbitrarily long
times, which is tighter than the corresponding state-independent bounds set by
known speed limits. This bound further generalizes naturally to the real-time
dynamics of time-dependent or dissipative systems where no energy spectrum
exists. We use this result to constrain the scrambling of information in
interacting many-body systems. For Hamiltonian systems, we show that the
fundamental question of the fastest possible scrambling time -- without any
restrictions on the structure of interactions -- maps to a purely mathematical
property of the density of states involving the non-negativity of Fourier
transforms. We illustrate these bounds in the Sachdev-Ye-Kitaev model, where we
show that despite its "maximally chaotic" nature, the sustained scrambling of
sufficiently large fermion subsystems via entanglement generation requires an
exponentially long time in the subsystem size.
