  Large-scale, high-quality corpora are critical for advancing research in
coreference resolution. However, existing datasets vary in their definition of
coreferences and have been collected via complex and lengthy guidelines that
are curated for linguistic experts. These concerns have sparked a growing
interest among researchers to curate a unified set of guidelines suitable for
annotators with various backgrounds. In this work, we develop a
crowdsourcing-friendly coreference annotation methodology, ezCoref, consisting
of an annotation tool and an interactive tutorial. We use ezCoref to
re-annotate 240 passages from seven existing English coreference datasets
(spanning fiction, news, and multiple other domains) while teaching annotators
only cases that are treated similarly across these datasets. Surprisingly, we
find that reasonable quality annotations were already achievable (>90%
agreement between the crowd and expert annotations) even without extensive
training. On carefully analyzing the remaining disagreements, we identify the
presence of linguistic cases that our annotators unanimously agree upon but
lack unified treatments (e.g., generic pronouns, appositives) in existing
datasets. We propose the research community should revisit these phenomena when
curating future unified annotation guidelines.
