  Large language models (LLMs) excel in most NLP tasks but also require
expensive cloud servers for deployment due to their size, while smaller models
that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in
terms of response quality. Therefore in this work we propose a hybrid inference
approach which combines their respective strengths to save cost and maintain
quality. Our approach uses a router that assigns queries to the small or large
model based on the predicted query difficulty and the desired quality level.
The desired quality level can be tuned dynamically at test time to seamlessly
trade quality for cost as per the scenario requirements. In experiments our
approach allows us to make up to 40% fewer calls to the large model, with no
drop in response quality.
