  The problem of attribution is concerned with identifying the parts of an
input that are responsible for a model's output. An important family of
attribution methods is based on measuring the effect of perturbations applied
to the input. In this paper, we discuss some of the shortcomings of existing
approaches to perturbation analysis and address them by introducing the concept
of extremal perturbations, which are theoretically grounded and interpretable.
We also introduce a number of technical innovations to compute extremal
perturbations, including a new area constraint and a parametric family of
smooth perturbations, which allow us to remove all tunable hyper-parameters
from the optimization problem. We analyze the effect of perturbations as a
function of their area, demonstrating excellent sensitivity to the spatial
properties of the deep neural network under stimulation. We also extend
perturbation analysis to the intermediate layers of a network. This application
allows us to identify the salient channels necessary for classification, which,
when visualized using feature inversion, can be used to elucidate model
behavior. Lastly, we introduce TorchRay, an interpretability library built on
PyTorch.
