  The recent advancements in large language models (LLMs) have been
extraordinary, yet the escalating inference costs associated with them present
challenges in real-world applications. To address these challenges, we propose
a novel approach called Early-exiting Speculative Decoding (EESD) with lossless
acceleration. Specifically, EESD utilizes a segment of the LLM to generate
draft tokens, incorporating Early-exiting structures after the first N layers.
To enhance the quality of draft tokens, a self-distillation method is
integrated. This early-exiting design not only reduces deployment and training
costs but also significantly accelerates the token generation speed. Moreover,
we introduce a novel sampling mechanism that leverages Thompson Sampling to
regulate the generation processes, automatically determining the quantity of
draft tokens in each round. The original LLM is then employed to validate these
draft tokens through a single forward pass, and thus guarantees that the final
output text maintains a distribution consistent with vanilla auto-regressive
decoding. The experimental results on both 13B and 70B models demonstrate that
our approach decodes tokens at a markedly accelerated rate compared to prior
methods, showing the effectiveness of our approach.
