  Machine learning models, including state-of-the-art deep neural networks, are
vulnerable to small perturbations that cause unexpected classification errors.
This unexpected lack of robustness raises fundamental questions about their
generalization properties and poses a serious concern for practical
deployments. As such perturbations can remain imperceptible - the formed
adversarial examples demonstrate an inherent inconsistency between vulnerable
machine learning models and human perception - some prior work casts this
problem as a security issue. Despite the significance of the discovered
instabilities and ensuing research, their cause is not well understood and no
effective method has been developed to address the problem. In this paper, we
present a novel theory to explain why this unpleasant phenomenon exists in deep
neural networks. Based on that theory, we introduce a simple, efficient, and
effective training approach, Batch Adjusted Network Gradients (BANG), which
significantly improves the robustness of machine learning models. While the
BANG technique does not rely on any form of data augmentation or the
utilization of adversarial images for training, the resultant classifiers are
more resistant to adversarial perturbations while maintaining or even enhancing
the overall classification performance.
