  Established cross-document coreference resolution (CDCR) datasets contain
event-centric coreference chains of events and entities with identity
relations. These datasets establish strict definitions of the coreference
relations across related tests but typically ignore anaphora with more vague
context-dependent loose coreference relations. In this paper, we qualitatively
and quantitatively compare the annotation schemes of ECB+, a CDCR dataset with
identity coreference relations, and NewsWCL50, a CDCR dataset with a mix of
loose context-dependent and strict coreference relations. We propose a phrasing
diversity metric (PD) that encounters for the diversity of full phrases unlike
the previously proposed metrics and allows to evaluate lexical diversity of the
CDCR datasets in a higher precision. The analysis shows that coreference chains
of NewsWCL50 are more lexically diverse than those of ECB+ but annotating of
NewsWCL50 leads to the lower inter-coder reliability. We discuss the different
tasks that both CDCR datasets create for the CDCR models, i.e., lexical
disambiguation and lexical diversity. Finally, to ensure generalizability of
the CDCR models, we propose a direction for CDCR evaluation that combines CDCR
datasets with multiple annotation schemes that focus of various properties of
the coreference chains.
