  Efficiency is a key property to foster inclusiveness and reduce environmental
costs, especially in an era of LLMs. In this work, we provide a comprehensive
evaluation of efficiency for MT evaluation metrics. Our approach involves
replacing computation-intensive transformers with lighter alternatives and
employing linear and quadratic approximations for alignment algorithms on top
of LLM representations. We evaluate six (reference-free and reference-based)
metrics across three MT datasets and examine 16 lightweight transformers. In
addition, we look into the training efficiency of metrics like COMET by
utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal
balance between quality and efficiency, (b) CPU speed-ups are more substantial
than those on GPU; (c) WMD approximations yield no efficiency gains while
reducing quality and (d) adapters enhance training efficiency (regarding
backward pass speed and memory requirements) as well as, in some cases, metric
quality. These findings can help to strike a balance between evaluation speed
and quality, which is essential for effective NLG systems. Furthermore, our
research contributes to the ongoing efforts to optimize NLG evaluation metrics
with minimal impact on performance. To our knowledge, ours is the most
comprehensive analysis of different aspects of efficiency for MT metrics
conducted so far.
