  We consider the problem of pretraining a two-stage open-domain question
answering (QA) system (retriever + reader) with strong transfer capabilities.
The key challenge is how to construct a large amount of high-quality
question-answer-context triplets without task-specific annotations.
Specifically, the triplets should align well with downstream tasks by: (i)
covering a wide range of domains (for open-domain applications), (ii) linking a
question to its semantically relevant context with supporting evidence (for
training the retriever), and (iii) identifying the correct answer in the
context (for training the reader). Previous pretraining approaches generally
fall short of one or more of these requirements. In this work, we automatically
construct a large-scale corpus that meets all three criteria by consulting
millions of references cited within Wikipedia. The well-aligned pretraining
signals benefit both the retriever and the reader significantly. Our pretrained
retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our
pretrained reader, the entire system improves by up to 4% in exact match.
