  Deep neural networks (DNNs) have set benchmarks on a wide array of supervised
learning tasks. Trained DNNs, however, often lack robustness to minor
adversarial perturbations to the input, which undermines their true
practicality. Recent works have increased the robustness of DNNs by fitting
networks using adversarially-perturbed training samples, but the improved
performance can still be far below the performance seen in non-adversarial
settings. A significant portion of this gap can be attributed to the decrease
in generalization performance due to adversarial training. In this work, we
extend the notion of margin loss to adversarial settings and bound the
generalization error for DNNs trained under several well-known gradient-based
attack schemes, motivating an effective regularization scheme based on spectral
normalization of the DNN's weight matrices. We also provide a
computationally-efficient method for normalizing the spectral norm of
convolutional layers with arbitrary stride and padding schemes in deep
convolutional networks. We evaluate the power of spectral normalization
extensively on combinations of datasets, network architectures, and adversarial
training schemes. The code is available at
https://github.com/jessemzhang/dl_spectral_normalization.
