  Measuring the performance of natural language processing models is
challenging. Traditionally used metrics, such as BLEU and ROUGE, originally
devised for machine translation and summarization, have been shown to suffer
from low correlation with human judgment and a lack of transferability to other
tasks and languages. In the past 15 years, a wide range of alternative metrics
have been proposed. However, it is unclear to what extent this has had an
impact on NLP benchmarking efforts. Here we provide the first large-scale
cross-sectional analysis of metrics used for measuring performance in natural
language processing. We curated, mapped and systematized more than 3500 machine
learning model performance results from the open repository 'Papers with Code'
to enable a global and comprehensive analysis. Our results suggest that the
large majority of natural language processing metrics currently used have
properties that may result in an inadequate reflection of a models'
performance. Furthermore, we found that ambiguities and inconsistencies in the
reporting of metrics may lead to difficulties in interpreting and comparing
model performances, impairing transparency and reproducibility in NLP research.
