  Cascades and speculative decoding are two common approaches to improving
language models' inference efficiency. Both approaches involve interleaving
models of different sizes, but via fundamentally distinct mechanisms: cascades
employ a deferral rule that invokes the larger model only for "hard" inputs,
while speculative decoding uses speculative execution to primarily invoke the
larger model in parallel verification mode. These mechanisms offer different
benefits: empirically, cascades are often capable of yielding better quality
than even the larger model, while theoretically, speculative decoding offers a
guarantee of quality-neutrality. In this paper, we leverage the best of both
these approaches by designing new speculative cascading techniques that
implement their deferral rule through speculative execution. We characterize
the optimal deferral rule for our speculative cascades, and employ a plug-in
approximation to the optimal rule. Through experiments with T5 models on
benchmark language tasks, we show that the proposed approach yields better
cost-quality trade-offs than cascading and speculative decoding baselines.
