  Deployment of autoregressive large language models (LLMs) is costly, and as
these models increase in size, the associated costs will become even more
considerable. Consequently, different methods have been proposed to accelerate
the token generation process and reduce costs. Speculative decoding (SD) is
among the most promising approaches to speed up the LLM decoding process by
verifying multiple tokens in parallel and using an auxiliary smaller draft
model to generate the possible tokens. In SD, usually, one draft model is used
to serve a specific target model; however, in practice, LLMs are diverse, and
we might need to deal with many target models or more than one target model
simultaneously. In this scenario, it is not clear which draft model should be
used for which target model, and searching among different draft models or
training customized draft models can further increase deployment costs. In this
paper, we first introduce a novel multi-target scenario for the deployment of
draft models for faster inference. Then, we present a novel, more efficient
sorted speculative decoding mechanism that outperforms regular baselines in
multi-target settings. We evaluated our method on Spec-Bench in different
settings, including base models such as Vicuna 7B, 13B, and LLama Chat 70B. Our
results suggest that our draft models perform better than baselines for
multiple target models at the same time.
