  We explore a human-driven approach to annotation, curated training (CT), in
which annotation is framed as teaching the system by using interactive search
to identify informative snippets of text to annotate, unlike traditional
approaches which either annotate preselected text or use active learning. A
trained annotator performed 80 hours of CT for the thirty event types of the
NIST TAC KBP Event Argument Extraction evaluation. Combining this annotation
with ACE results in a 6% reduction in error and the learning curve of CT
plateaus more slowly than for full-document annotation. 3 NLP researchers
performed CT for one event type and showed much sharper learning curves with
all three exceeding ACE performance in less than ninety minutes, suggesting
that CT can provide further benefits when the annotator deeply understands the
system.
