  Many state-of-the-art natural language understanding (NLU) models are based
on pretrained neural language models. These models often make inferences using
information from multiple sources. An important class of such inferences are
those that require both background knowledge, presumably contained in a model's
pretrained parameters, and instance-specific information that is supplied at
inference time. However, the integration and reasoning abilities of NLU models
in the presence of multiple knowledge sources have been largely understudied.
In this work, we propose a test suite of coreference resolution subtasks that
require reasoning over multiple facts. These subtasks differ in terms of which
knowledge sources contain the relevant facts. We also introduce subtasks where
knowledge is present only at inference time using fictional knowledge. We
evaluate state-of-the-art coreference resolution models on our dataset. Our
results indicate that several models struggle to reason on-the-fly over
knowledge observed both at pretrain time and at inference time. However, with
task-specific training, a subset of models demonstrates the ability to
integrate certain knowledge types from multiple sources. Still, even the best
performing models seem to have difficulties with reliably integrating knowledge
presented only at inference time.
