  Due to the significant advancement of Natural Language Processing and
Computer Vision-based models, Visual Question Answering (VQA) systems are
becoming more intelligent and advanced. However, they are still error-prone
when dealing with relatively complex questions. Therefore, it is important to
understand the behaviour of the VQA models before adopting their results. In
this paper, we introduce an interpretability approach for VQA models by
generating counterfactual images. Specifically, the generated image is supposed
to have the minimal possible change to the original image and leads the VQA
model to give a different answer. In addition, our approach ensures that the
generated image is realistic. Since quantitative metrics cannot be employed to
evaluate the interpretability of the model, we carried out a user study to
assess different aspects of our approach. In addition to interpreting the
result of VQA models on single images, the obtained results and the discussion
provides an extensive explanation of VQA models' behaviour.
