  Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA
task that requires retrieval of external knowledge to answer questions about
images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve
documents from external knowledge bases, such as Wikipedia, but with DPR
trained separately from answer generation, introducing a potential limit on the
overall system performance. Instead, we propose a joint training scheme which
includes differentiable DPR integrated with answer generation so that the
system can be trained in an end-to-end fashion. Our experiments show that our
scheme outperforms recent OK-VQA systems with strong DPR for retrieval. We also
introduce new diagnostic metrics to analyze how retrieval and generation
interact. The strong retrieval ability of our model significantly reduces the
number of retrieved documents needed in training, yielding significant benefits
in answer quality and computation required for training.
