  Despite the remarkable strides made by autoregressive language models, their
potential is often hampered by the slow inference speeds inherent in sequential
token generation. Blockwise parallel decoding (BPD) was proposed by Stern et
al. as a method to improve inference speed of language models by simultaneously
predicting multiple future tokens, termed block drafts, which are subsequently
verified and conditionally accepted by the autoregressive model. This paper
contributes to the understanding and improvement of block drafts in two ways.
First, we analyze the token distributions produced by multiple prediction
heads. Secondly, we leverage this analysis to develop algorithms to improve BPD
inference speed by refining the block drafts using n-gram and neural language
models. Experiments demonstrate that refined block drafts yield a +5-21%
increase in block efficiency (i.e., the number of accepted tokens from the
block draft) across diverse datasets.
