  With the increasingly giant scales of (causal) large language models (LLMs),
the inference efficiency comes as one of the core concerns along the improved
performance. In contrast to the memory footprint, the latency bottleneck seems
to be of greater importance as there can be billions of requests to a LLM
(e.g., GPT-4) per day. The bottleneck is mainly due to the autoregressive
innateness of LLMs, where tokens can only be generated sequentially during
decoding. To alleviate the bottleneck, the idea of speculative execution, which
originates from the field of computer architecture, is introduced to LLM
decoding in a \textit{draft-then-verify} style. Under this regime, a sequence
of tokens will be drafted in a fast pace by utilizing some heuristics, and then
the tokens shall be verified in parallel by the LLM. As the costly sequential
inference is parallelized, LLM decoding speed can be significantly boosted.
Driven by the success of LLMs in recent couple of years, a growing literature
in this direction has emerged. Yet, there lacks a position survey to summarize
the current landscape and draw a roadmap for future development of this
promising area. To meet this demand, we present the very first survey paper
that reviews and unifies literature of speculative execution in LLMs (e.g.,
blockwise parallel decoding, speculative decoding, etc.) in a comprehensive
framework and a systematic taxonomy. Based on the taxonomy, we present a
critical review and comparative analysis of the current arts. Finally we
highlight various key challenges and future directions to further develop the
area.
