  Normalization techniques have become a basic component in modern
convolutional neural networks (ConvNets). In particular, many recent works
demonstrate that promoting the orthogonality of the weights helps train deep
models and improve robustness. For ConvNets, most existing methods are based on
penalizing or normalizing weight matrices derived from concatenating or
flattening the convolutional kernels. These methods often destroy or ignore the
benign convolutional structure of the kernels; therefore, they are often
expensive or impractical for deep ConvNets. In contrast, we introduce a simple
and efficient "Convolutional Normalization" (ConvNorm) method that can fully
exploit the convolutional structure in the Fourier domain and serve as a simple
plug-and-play module to be conveniently incorporated into any ConvNets. Our
method is inspired by recent work on preconditioning methods for convolutional
sparse coding and can effectively promote each layer's channel-wise isometry.
Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm
of the weight matrices and hence improve the Lipschitzness of the network,
leading to easier training and improved robustness for deep ConvNets. Applied
to classification under noise corruptions and generative adversarial network
(GAN), we show that the ConvNorm improves the robustness of common ConvNets
such as ResNet and the performance of GAN. We verify our findings via numerical
experiments on CIFAR and ImageNet.
