  Saliency Map, the gradient of the score function with respect to the input,
is the most basic technique for interpreting deep neural network decisions.
However, saliency maps are often visually noisy. Although several hypotheses
were proposed to account for this phenomenon, there are few works that provide
rigorous analyses of noisy saliency maps. In this paper, we firstly propose a
new hypothesis that noise may occur in saliency maps when irrelevant features
pass through ReLU activation functions. Then, we propose Rectified Gradient, a
method that alleviates this problem through layer-wise thresholding during
backpropagation. Experiments with neural networks trained on CIFAR-10 and
ImageNet showed effectiveness of our method and its superiority to other
attribution methods.
