  The recent emergence of Large Language Models based on the Transformer
architecture has enabled dramatic advancements in the field of Natural Language
Processing. However, these models have long inference latency, which limits
their deployment and makes them prohibitively expensive for various real-time
applications. The inference latency is further exacerbated by autoregressive
generative tasks, as models need to run iteratively to generate tokens
sequentially without leveraging token-level parallelization. To address this,
we propose Big Little Decoder (BiLD), a framework that can improve inference
efficiency and latency for a wide range of text generation applications. The
BiLD framework contains two models with different sizes that collaboratively
generate text. The small model runs autoregressively to generate text with a
low inference cost, and the large model is only invoked occasionally to refine
the small model's inaccurate predictions in a non-autoregressive manner. To
coordinate the small and large models, BiLD introduces two simple yet effective
policies: (1) the fallback policy that determines when to hand control over to
the large model; and (2) the rollback policy that determines when the large
model needs to correct the small model's inaccurate predictions. To evaluate
our framework across different tasks and models, we apply BiLD to various text
generation scenarios encompassing machine translation on IWSLT 2017 De-En and
WMT 2014 De-En, and summarization on XSUM and CNN/DailyMail. On an NVIDIA T4
GPU, our framework achieves a speedup of up to 2.12x speedup with minimal
generation quality degradation. Furthermore, our framework is fully
plug-and-play and can be applied without any modifications in the training
process or model architecture. Our code is open-sourced
