  Visual Question Answering (VQA) in its ideal form lets us study reasoning in
the joint space of vision and language and serves as a proxy for the AI task of
scene understanding. However, most VQA benchmarks to date are focused on
questions such as simple counting, visual attributes, and object detection that
do not require reasoning or knowledge beyond what is in the image. In this
paper, we address the task of knowledge-based visual question answering and
provide a benchmark, called OK-VQA, where the image content is not sufficient
to answer the questions, encouraging methods that rely on external knowledge
resources. Our new dataset includes more than 14,000 questions that require
external knowledge to answer. We show that the performance of the
state-of-the-art VQA models degrades drastically in this new setting. Our
analysis shows that our knowledge-based VQA task is diverse, difficult, and
large compared to previous knowledge-based VQA datasets. We hope that this
dataset enables researchers to open up new avenues for research in this domain.
See http://okvqa.allenai.org to download and browse the dataset.
