  This work presents a novel four-stage open-domain QA pipeline R2-D2 (Rank
twice, reaD twice). The pipeline is composed of a retriever, passage reranker,
extractive reader, generative reader and a mechanism that aggregates the final
prediction from all system's components. We demonstrate its strength across
three open-domain QA datasets: NaturalQuestions, TriviaQA and EfficientQA,
surpassing state-of-the-art on the first two. Our analysis demonstrates that:
(i) combining extractive and generative reader yields absolute improvements up
to 5 exact match and it is at least twice as effective as the posterior
averaging ensemble of the same models with different parameters, (ii) the
extractive reader with fewer parameters can match the performance of the
generative reader on extractive QA datasets.
