  We present a method to represent input texts by contextualizing them jointly
with dynamically retrieved textual encyclopedic background knowledge from
multiple documents. We apply our method to reading comprehension tasks by
encoding questions and passages together with background sentences about the
entities they mention. We show that integrating background knowledge from text
is effective for tasks focusing on factual reasoning and allows direct reuse of
powerful pretrained BERT-style encoders. Moreover, knowledge integration can be
further improved with suitable pretraining via a self-supervised masked
language model objective over words in background-augmented input text. On
TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable
RoBERTa models which do not integrate background knowledge dynamically. On
MRQA, a large collection of diverse QA datasets, we see consistent gains
in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2
F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).
