  We show new connections between adversarial learning and explainability for
deep neural networks (DNNs). One form of explanation of the output of a neural
network model in terms of its input features, is a vector of
feature-attributions. Two desirable characteristics of an attribution-based
explanation are: (1) $\textit{sparseness}$: the attributions of irrelevant or
weakly relevant features should be negligible, thus resulting in
$\textit{concise}$ explanations in terms of the significant features, and (2)
$\textit{stability}$: it should not vary significantly within a small local
neighborhood of the input. Our first contribution is a theoretical exploration
of how these two properties (when using attributions based on Integrated
Gradients, or IG) are related to adversarial training, for a class of 1-layer
networks (which includes logistic regression models for binary and multi-class
classification); for these networks we show that (a) adversarial training using
an $\ell_\infty$-bounded adversary produces models with sparse attribution
vectors, and (b) natural model-training while encouraging stable explanations
(via an extra term in the loss function), is equivalent to adversarial
training. Our second contribution is an empirical verification of phenomenon
(a), which we show, somewhat surprisingly, occurs $\textit{not only}$
$\textit{in 1-layer networks}$, $\textit{but also DNNs}$ $\textit{trained on }$
$\textit{standard image datasets}$, and extends beyond IG-based attributions,
to those based on DeepSHAP: adversarial training with $\ell_\infty$-bounded
perturbations yields significantly sparser attribution vectors, with little
degradation in performance on natural test data, compared to natural training.
Moreover, the sparseness of the attribution vectors is significantly better
than that achievable via $\ell_1$-regularized natural training.
