  Visual Question Answering (VQA) methods have made incredible progress, but
suffer from a failure to generalize. This is visible in the fact that they are
vulnerable to learning coincidental correlations in the data rather than deeper
relations between image content and ideas expressed in language. We present a
dataset that takes a step towards addressing this problem in that it contains
questions expressed in two languages, and an evaluation process that co-opts a
well understood image-based metric to reflect the method's ability to reason.
Measuring reasoning directly encourages generalization by penalizing answers
that are coincidentally correct. The dataset reflects the scene-text version of
the VQA problem, and the reasoning evaluation can be seen as a text-based
version of a referring expression challenge. Experiments and analysis are
provided that show the value of the dataset.
