  SOTA coreference resolution produces increasingly impressive scores on the
OntoNotes benchmark. However lack of comparable data following the same scheme
for more genres makes it difficult to evaluate generalizability to open domain
data. This paper provides a dataset and comprehensive evaluation showing that
the latest neural LM based end-to-end systems degrade very substantially out of
domain. We make an OntoNotes-like coreference dataset called OntoGUM publicly
available, converted from GUM, an English corpus covering 12 genres, using
deterministic rules, which we evaluate. Thanks to the rich syntactic and
discourse annotations in GUM, we are able to create the largest human-annotated
coreference corpus following the OntoNotes guidelines, and the first to be
evaluated for consistency with the OntoNotes scheme. Out-of-domain evaluation
across 12 genres shows nearly 15-20% degradation for both deterministic and
deep learning systems, indicating a lack of generalizability or covert
overfitting in existing coreference resolution models.
