  A precise understanding of why units in an artificial network respond to
certain stimuli would constitute a big step towards explainable artificial
intelligence. One widely used approach towards this goal is to visualize unit
responses via activation maximization. These synthetic feature visualizations
are purported to provide humans with precise information about the image
features that cause a unit to be activated - an advantage over other
alternatives like strongly activating natural dataset samples. If humans indeed
gain causal insight from visualizations, this should enable them to predict the
effect of an intervention, such as how occluding a certain patch of the image
(say, a dog's head) changes a unit's activation. Here, we test this hypothesis
by asking humans to decide which of two square occlusions causes a larger
change to a unit's activation. Both a large-scale crowdsourced experiment and
measurements with experts show that on average the extremely activating feature
visualizations by Olah et al. (2017) indeed help humans on this task ($68 \pm
4$% accuracy; baseline performance without any visualizations is $60 \pm 3$%).
However, they do not provide any substantial advantage over other
visualizations (such as e.g. dataset samples), which yield similar performance
($66\pm3$% to $67 \pm3$% accuracy). Taken together, we propose an objective
psychophysical task to quantify the benefit of unit-level interpretability
methods for humans, and find no evidence that a widely-used feature
visualization method provides humans with better "causal understanding" of unit
activations than simple alternative visualizations.
