  Large Language Models (LLMs) have shown impressive results on a variety of
text understanding tasks. Search queries though pose a unique challenge, given
their short-length and lack of nuance or context. Complicated feature
engineering efforts do not always lead to downstream improvements as their
performance benefits may be offset by increased complexity of knowledge
distillation. Thus, in this paper we make the following contributions: (1) We
demonstrate that Retrieval Augmentation of queries provides LLMs with valuable
additional context enabling improved understanding. While Retrieval
Augmentation typically increases latency of LMs (thus hurting distillation
efficacy), (2) we provide a practical and effective way of distilling Retrieval
Augmentation LLMs. Specifically, we use a novel two-stage distillation approach
that allows us to carry over the gains of retrieval augmentation, without
suffering the increased compute typically associated with it. (3) We
demonstrate the benefits of the proposed approach (QUILL) on a billion-scale,
real-world query understanding system resulting in huge gains. Via extensive
experiments, including on public benchmarks, we believe this work offers a
recipe for practical use of retrieval-augmented query understanding.
