  A variety of methods have been proposed to try to explain how deep neural
networks make their decisions. Key to those approaches is the need to sample
the pixel space efficiently in order to derive importance maps. However, it has
been shown that the sampling methods used to date introduce biases and other
artifacts, leading to inaccurate estimates of the importance of individual
pixels and severely limit the reliability of current explainability methods.
Unfortunately, the alternative -- to exhaustively sample the image space is
computationally prohibitive. In this paper, we introduce EVA (Explaining using
Verified perturbation Analysis) -- the first explainability method guarantee to
have an exhaustive exploration of a perturbation space. Specifically, we
leverage the beneficial properties of verified perturbation analysis -- time
efficiency, tractability and guaranteed complete coverage of a manifold -- to
efficiently characterize the input variables that are most likely to drive the
model decision. We evaluate the approach systematically and demonstrate
state-of-the-art results on multiple benchmarks.
