  Large language models (LLM) have recently attracted surging interest due to
their outstanding capabilities across various domains. However, enabling
efficient LLM inference is challenging due to its autoregressive decoding that
generates tokens only one at a time. Although research works apply pruning or
quantization to speed up LLM inference, they typically require fine-tuning the
LLM, incurring significant time and economic costs. Meanwhile, speculative
decoding has been proposed to use small speculative models (SSMs) to accelerate
the inference of LLM. However, the low acceptance rate of SSM and the high
verification cost of LLM prohibit further performance improvement of inference.
In this paper, we propose Minions, an LLM inference system that accelerates LLM
inference with a collective and adaptive speculative generation. Specifically,
Minions proposes a majority-voted mechanism to leverage multiple SSMs to
jointly speculate the outputs of LLM, which improves the inference performance
without introducing prohibitive computation costs for LLM. To better trade off
the number of tokens speculated from SSM and the verification cost of LLM,
Minions proposes an adaptive mechanism to dynamically determine the optimal
speculation length of SSM, which can achieve better inference performance
across different models, datasets, and hyper-parameters. In addition, Minions
decouples the SSM decoding and LLM verification efficiently and adopts a
pipelined execution mechanism to further improve the inference performance of
LLM. By comparing with the state-of-the-art LLM inference systems, we
demonstrate that Minions can achieve higher inference throughput and lower
inference time.
