  Document-level information extraction (IE) tasks have recently begun to be
revisited in earnest using the end-to-end neural network techniques that have
been successful on their sentence-level IE counterparts. Evaluation of the
approaches, however, has been limited in a number of dimensions. In particular,
the precision/recall/F1 scores typically reported provide few insights on the
range of errors the models make. We build on the work of Kummerfeld and Klein
(2013) to propose a transformation-based framework for automating error
analysis in document-level event and (N-ary) relation extraction. We employ our
framework to compare two state-of-the-art document-level template-filling
approaches on datasets from three domains; and then, to gauge progress in IE
since its inception 30 years ago, vs. four systems from the MUC-4 (1992)
evaluation.
