  The complex compositional structure of language makes problems at the
intersection of vision and language challenging. But language also provides a
strong prior that can result in good superficial performance, without the
underlying models truly understanding the visual content. This can hinder
progress in pushing state of art in the computer vision aspects of multi-modal
AI. In this paper, we address binary Visual Question Answering (VQA) on
abstract scenes. We formulate this problem as visual verification of concepts
inquired in the questions. Specifically, we convert the question to a tuple
that concisely summarizes the visual concept to be detected in the image. If
the concept can be found in the image, the answer to the question is "yes", and
otherwise "no". Abstract scenes play two roles (1) They allow us to focus on
the high-level semantics of the VQA task as opposed to the low-level
recognition problems, and perhaps more importantly, (2) They provide us the
modality to balance the dataset such that language priors are controlled, and
the role of vision is essential. In particular, we collect fine-grained pairs
of scenes for every question, such that the answer to the question is "yes" for
one scene, and "no" for the other for the exact same question. Indeed, language
priors alone do not perform better than chance on our balanced dataset.
Moreover, our proposed approach matches the performance of a state-of-the-art
VQA approach on the unbalanced dataset, and outperforms it on the balanced
dataset.
