  Speculative decoding is commonly used for reducing the inference latency of
large language models. Its effectiveness depends highly on the speculation
lookahead (SL)-the number of tokens generated by the draft model at each
iteration. In this work we show that the common practice of using the same SL
for all iterations (static SL) is suboptimal. We introduce DISCO (DynamIc
SpeCulation lookahead Optimization), a novel method for dynamically selecting
the SL. Our experiments with four datasets show that DISCO reaches an average
speedup of 10% compared to the best static SL baseline, while generating the
exact same text.
