  In open-domain question answering, a model receives a text question as input
and searches for the correct answer using a large evidence corpus. The
retrieval step is especially difficult as typical evidence corpora have
\textit{millions} of documents, each of which may or may not have the correct
answer to the question. Very recently, dense models have replaced sparse
methods as the de facto retrieval method. Rather than focusing on lexical
overlap to determine similarity, dense methods build an encoding function that
captures semantic similarity by learning from a small collection of
question-answer or question-context pairs. In this paper, we investigate dense
retrieval models in the context of open-domain question answering across
different input distributions. To do this, first we introduce an entity-rich
question answering dataset constructed from Wikidata facts and demonstrate
dense models are unable to generalize to unseen input question distributions.
Second, we perform analyses aimed at better understanding the source of the
problem and propose new training techniques to improve out-of-domain
performance on a wide variety of datasets. We encourage the field to further
investigate the creation of a single, universal dense retrieval model that
generalizes well across all input distributions.
