  Recent advances in Transformer-based large language models (LLMs) have led to
significant performance improvements across many tasks. These gains come with a
drastic increase in the models' size, potentially leading to slow and costly
use at inference time. In practice, however, the series of generations made by
LLMs is composed of varying levels of difficulty. While certain predictions
truly benefit from the models' full capacity, other continuations are more
trivial and can be solved with reduced compute. In this work, we introduce
Confident Adaptive Language Modeling (CALM), a framework for dynamically
allocating different amounts of compute per input and generation timestep.
Early exit decoding involves several challenges that we address here, such as:
(1) what confidence measure to use; (2) connecting sequence-level constraints
to local per-token exit decisions; and (3) attending back to missing hidden
representations due to early exits in previous tokens. Through theoretical
analysis and empirical experiments on three diverse text generation tasks, we
demonstrate the efficacy of our framework in reducing compute -- potential
speedup of up to $\times 3$ -- while provably maintaining high performance.
