  Natural language processing (NLP) systems are increasingly trained to
generate open-ended text rather than classifying between responses. This makes
research on evaluation metrics for generated language -- functions that score
system output given the context and/or human reference responses -- of critical
importance. However, different metrics have different strengths and biases, and
reflect human intuitions better on some tasks than others. There is currently
no simple, unified way to compare, analyse or evaluate metrics across a
representative set of tasks. Here, we describe the Benchmark to Evaluate
Automatic Metrics (BEAMetrics), a resource to make research into new metrics
itself easier to evaluate. BEAMetrics users can quickly compare existing and
new metrics with human judgements across a diverse set of tasks, quality
dimensions (fluency vs. coherence vs. informativeness etc), and languages. As
generation experts might predict, BEAMetrics reveals stark task-dependent
differences between existing metrics, and consistently poor performance on
tasks with complex answer spaces or high reliance on general knowledge. While
this analysis highlights a critical issue facing current research practice,
BEAMetrics also contribute to its resolution by facilitating research into
better metrics -- particularly those that can account for the complex
interaction between context and general knowledge inherent to many modern NLP
applications. BEAMetrics is available under the MIT License:
https://github.com/ThomasScialom/BEAMetrics
