  A visual counterfactual explanation replaces image regions in a query image
with regions from a distractor image such that the system's decision on the
transformed image changes to the distractor class. In this work, we present a
novel framework for computing visual counterfactual explanations based on two
key ideas. First, we enforce that the replaced and replacer regions contain the
same semantic part, resulting in more semantically consistent explanations.
Second, we use multiple distractor images in a computationally efficient way
and obtain more discriminative explanations with fewer region replacements. Our
approach is 27 % more semantically consistent and an order of magnitude faster
than a competing method on three fine-grained image recognition datasets. We
highlight the utility of our counterfactuals over existing works through
machine teaching experiments where we teach humans to classify different bird
species. We also complement our explanations with the vocabulary of parts and
attributes that contributed the most to the system's decision. In this task as
well, we obtain state-of-the-art results when using our counterfactual
explanations relative to existing works, reinforcing the importance of
semantically consistent explanations. Source code is available at
https://github.com/facebookresearch/visual-counterfactuals.
