  Most Outside-Knowledge Visual Question Answering (OK-VQA) systems employ a
two-stage framework that first retrieves external knowledge given the visual
question and then predicts the answer based on the retrieved content. However,
the retrieved knowledge is often inadequate. Retrievals are frequently too
general and fail to cover specific knowledge needed to answer the question.
Also, the naturally available supervision (whether the passage contains the
correct answer) is weak and does not guarantee question relevancy. To address
these issues, we propose an Entity-Focused Retrieval (EnFoRe) model that
provides stronger supervision during training and recognizes question-relevant
entities to help retrieve more specific knowledge. Experiments show that our
EnFoRe model achieves superior retrieval performance on OK-VQA, the currently
largest outside-knowledge VQA dataset. We also combine the retrieved knowledge
with state-of-the-art VQA models, and achieve a new state-of-the-art
performance on OK-VQA.
