  Fast and reliable evaluation metrics are key to R&D progress. While
traditional natural language generation metrics are fast, they are not very
reliable. Conversely, new metrics based on large pretrained language models are
much more reliable, but require significant computational resources. In this
paper, we propose FrugalScore, an approach to learn a fixed, low cost version
of any expensive NLG metric, while retaining most of its original performance.
Experiments with BERTScore and MoverScore on summarization and translation show
that FrugalScore is on par with the original metrics (and sometimes better),
while having several orders of magnitude less parameters and running several
times faster. On average over all learned metrics, tasks, and variants,
FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35
times less parameters than the original metrics. We make our trained metrics
publicly available, to benefit the entire NLP community and in particular
researchers and practitioners with limited resources.
