  Existing large language models have to run K times to generate a sequence of
K tokens. In this paper, we present RecycleGPT, a generative language model
with fast decoding speed by recycling pre-generated model states without
running the whole model in multiple steps. Our approach relies on the
observation that adjacent tokens in a sequence usually have strong correlations
and the next token in a sequence can be reasonably guessed or inferred based on
the preceding ones. Experiments and analysis demonstrate the effectiveness of
our approach in lowering inference latency, achieving up to 1.4x speedup while
preserving high performance.
