  Deep learning algorithms have shown promising results in visual question
answering (VQA) tasks, but a more careful look reveals that they often do not
understand the rich signal they are being fed with. To understand and better
measure the generalization capabilities of VQA systems, we look at their
robustness to counterfactually augmented data. Our proposed augmentations are
designed to make a focused intervention on a specific property of the question
such that the answer changes. Using these augmentations, we propose a new
robustness measure, Robustness to Augmented Data (RAD), which measures the
consistency of model predictions between original and augmented examples.
Through extensive experimentation, we show that RAD, unlike classical accuracy
measures, can quantify when state-of-the-art systems are not robust to
counterfactuals. We find substantial failure cases which reveal that current
VQA systems are still brittle. Finally, we connect between robustness and
generalization, demonstrating the predictive power of RAD for performance on
unseen augmentations.
