  Scaling Visual Question Answering (VQA) to the open-domain and multi-hop
nature of web searches, requires fundamental advances in visual representation
learning, knowledge aggregation, and language generation. In this work, we
introduce WebQA, a challenging new benchmark that proves difficult for
large-scale state-of-the-art models which lack language groundable visual
representations for novel objects and the ability to reason, yet trivial for
humans. WebQA mirrors the way humans use the web: 1) Ask a question, 2) Choose
sources to aggregate, and 3) Produce a fluent language response. This is the
behavior we should be expecting from IoT devices and digital assistants.
Existing work prefers to assume that a model can either reason about knowledge
in images or in text. WebQA includes a secondary text-only QA task to ensure
improved visual performance does not come at the cost of language
understanding. Our challenge for the community is to create unified multimodal
reasoning models that answer questions regardless of the source modality,
moving us closer to digital assistants that not only query language knowledge,
but also the richer visual online world.
