  Neural network interpretation methods, particularly feature attribution
methods, are known to be fragile with respect to adversarial input
perturbations. To address this, several methods for enhancing the local
smoothness of the gradient while training have been proposed for attaining
\textit{robust} feature attributions. However, the lack of considering the
normalization of the attributions, which is essential in their visualizations,
has been an obstacle to understanding and improving the robustness of feature
attribution methods. In this paper, we provide new insights by taking such
normalization into account. First, we show that for every non-negative
homogeneous neural network, a naive $\ell_2$-robust criterion for gradients is
\textit{not} normalization invariant, which means that two functions with the
same normalized gradient can have different values. Second, we formulate a
normalization invariant cosine distance-based criterion and derive its upper
bound, which gives insight for why simply minimizing the Hessian norm at the
input, as has been done in previous work, is not sufficient for attaining
robust feature attribution. Finally, we propose to combine both $\ell_2$ and
cosine distance-based criteria as regularization terms to leverage the
advantages of both in aligning the local gradient. As a result, we
experimentally show that models trained with our method produce much more
robust interpretations on CIFAR-10 and ImageNet-100 without significantly
hurting the accuracy, compared to the recent baselines. To the best of our
knowledge, this is the first work to verify the robustness of interpretation on
a larger-scale dataset beyond CIFAR-10, thanks to the computational efficiency
of our method.
