  In this paper we show that corpus-level aggregation hinders considerably the
capability of lexical metrics to accurately evaluate machine translation (MT)
systems. With empirical experiments we demonstrate that averaging individual
segment-level scores can make metrics such as BLEU and chrF correlate much
stronger with human judgements and make them behave considerably more similar
to neural metrics such as COMET and BLEURT. We show that this difference exists
because corpus- and segment-level aggregation differs considerably owing to the
classical average of ratio versus ratio of averages Mathematical problem.
Moreover, as we also show, such difference affects considerably the statistical
robustness of corpus-level aggregation. Considering that neural metrics
currently only cover a small set of sufficiently-resourced languages, the
results in this paper can help make the evaluation of MT systems for
low-resource languages more trustworthy.
