  Reducing serving cost and latency is a fundamental concern for the deployment
of language models (LMs) in business applications. To address this, cascades of
LMs offer an effective solution that conditionally employ smaller models for
simpler queries. Cascaded systems are typically built with independently
trained models, neglecting the advantages of considering inference-time
interactions of the cascaded LMs during training. In this paper, we present
cascade-aware training(CAT), an approach to optimizing the overall quality-cost
performance tradeoff of a cascade of LMs. We achieve inference-time benefits by
training the small LM with awareness of its place in a cascade and downstream
capabilities. We demonstrate the value of the proposed method with over 60 LM
tasks of the SuperGLUE, WMT22, and FLAN2021 datasets.
