  While Visual Question Answering (VQA) models continue to push the
state-of-the-art forward, they largely remain black-boxes - failing to provide
insight into how or why an answer is generated. In this ongoing work, we
propose addressing this shortcoming by learning to generate counterfactual
images for a VQA model - i.e. given a question-image pair, we wish to generate
a new image such that i) the VQA model outputs a different answer, ii) the new
image is minimally different from the original, and iii) the new image is
realistic. Our hope is that providing such counterfactual examples allows users
to investigate and understand the VQA model's internal mechanisms.
