  Open Information Extraction (OIE) is the task of extracting facts from
sentences in the form of relations and their corresponding arguments in
schema-free manner. Intrinsic performance of OIE systems is difficult to
measure due to the incompleteness of existing OIE benchmarks: the ground truth
extractions do not group all acceptable surface realizations of the same fact
that can be extracted from a sentence. To measure performance of OIE systems
more realistically, it is necessary to manually annotate complete facts (i.e.,
clusters of all acceptable surface realizations of the same fact) from input
sentences. We propose AnnIE: an interactive annotation platform that
facilitates such challenging annotation tasks and supports creation of complete
fact-oriented OIE evaluation benchmarks. AnnIE is modular and flexible in order
to support different use case scenarios (i.e., benchmarks covering different
types of facts). We use AnnIE to build two complete OIE benchmarks: one with
verb-mediated facts and another with facts encompassing named entities.
Finally, we evaluate several OIE systems on our complete benchmarks created
with AnnIE. Our results suggest that existing incomplete benchmarks are overly
lenient, and that OIE systems are not as robust as previously reported. We
publicly release AnnIE under non-restrictive license.
