  This paper describes and tests a method for carrying out quantified
reproducibility assessment (QRA) that is based on concepts and definitions from
metrology. QRA produces a single score estimating the degree of reproducibility
of a given system and evaluation measure, on the basis of the scores from, and
differences between, different reproductions. We test QRA on 18 system and
evaluation measure combinations (involving diverse NLP tasks and types of
evaluation), for each of which we have the original results and one to seven
reproduction results. The proposed QRA method produces
degree-of-reproducibility scores that are comparable across multiple
reproductions not only of the same, but of different original studies. We find
that the proposed method facilitates insights into causes of variation between
reproductions, and allows conclusions to be drawn about what changes to system
and/or evaluation design might lead to improved reproducibility.
