  Generative language models (LMs) have become omnipresent across data science.
For a wide variety of tasks, inputs can be phrased as natural language prompts
for an LM, from whose output the solution can then be extracted. LM performance
has consistently been increasing with model size - but so has the monetary cost
of querying the ever larger models. Importantly, however, not all inputs are
equally hard: some require larger LMs for obtaining a satisfactory solution,
whereas for others smaller LMs suffice. Based on this fact, we design a
framework for cost-effective language model choice, called "Fly-swat or cannon"
(FORC). Given a set of inputs and a set of candidate LMs, FORC judiciously
assigns each input to an LM predicted to do well on the input according to a
so-called meta-model, aiming to achieve high overall performance at low cost.
The cost-performance tradeoff can be flexibly tuned by the user. Options
include, among others, maximizing total expected performance (or the number of
processed inputs) while staying within a given cost budget, or minimizing total
cost while processing all inputs. We evaluate FORC on 14 datasets covering five
natural language tasks, using four candidate LMs of vastly different size and
cost. With FORC, we match the performance of the largest available LM while
achieving a cost reduction of 63%. Via our publicly available library,
researchers as well as practitioners can thus save large amounts of money
without sacrificing performance.
