  With the rapid development of NLP research, leaderboards have emerged as one
tool to track the performance of various systems on various NLP tasks. They are
effective in this goal to some extent, but generally present a rather
simplistic one-dimensional view of the submitted systems, communicated only
through holistic accuracy numbers. In this paper, we present a new
conceptualization and implementation of NLP evaluation: the ExplainaBoard,
which in addition to inheriting the functionality of the standard leaderboard,
also allows researchers to (i) diagnose strengths and weaknesses of a single
system (e.g.~what is the best-performing system bad at?) (ii) interpret
relationships between multiple systems. (e.g.~where does system A outperform
system B? What if we combine systems A, B, and C?) and (iii) examine prediction
results closely (e.g.~what are common errors made by multiple systems, or in
what contexts do particular errors occur?). So far, ExplainaBoard covers more
than 400 systems, 50 datasets, 40 languages, and 12 tasks. ExplainaBoard keeps
updated and is recently upgraded by supporting (1) multilingual multi-task
benchmark, (2) meta-evaluation, and (3) more complicated task: machine
translation, which reviewers also suggested.} We not only released an online
platform on the website \url{http://explainaboard.nlpedia.ai/} but also make
our evaluation tool an API with MIT Licence at Github
\url{https://github.com/neulab/explainaBoard} and PyPi
\url{https://pypi.org/project/interpret-eval/} that allows users to
conveniently assess their models offline. We additionally release all output
files from systems that we have run or collected to motivate "output-driven"
research in the future.
