  The robustness and anomaly detection capability of neural networks are
crucial topics for their safe adoption in the real-world. Moreover, the
over-parameterization of recent networks comes with high computational costs
and raises questions about its influence on robustness and anomaly detection.
In this work, we show that sparsity can make networks more robust and better
anomaly detectors. To motivate this even further, we show that a pre-trained
neural network contains, within its parameter space, sparse subnetworks that
are better at these tasks without any further training. We also show that
structured sparsity greatly helps in reducing the complexity of expensive
robustness and detection methods, while maintaining or even improving their
results on these tasks. Finally, we introduce a new method, SensNorm, which
uses the sensitivity of weights derived from an appropriate pruning method to
detect anomalous samples in the input.
