  This paper analyzes regularization terms proposed recently for improving the
adversarial robustness of deep neural networks (DNNs), from a theoretical point
of view. Specifically, we study possible connections between several effective
methods, including input-gradient regularization, Jacobian regularization,
curvature regularization, and a cross-Lipschitz functional. We investigate them
on DNNs with general rectified linear activations, which constitute one of the
most prevalent families of models for image classification and a host of other
machine learning applications. We shed light on essential ingredients of these
regularizations and re-interpret their functionality. Through the lens of our
study, more principled and efficient regularizations can possibly be invented
in the near future.
