  Since the Lipschitz properties of convolutional neural networks (CNNs) are
widely considered to be related to adversarial robustness, we theoretically
characterize the $\ell_1$ norm and $\ell_\infty$ norm of 2D multi-channel
convolutional layers and provide efficient methods to compute the exact
$\ell_1$ norm and $\ell_\infty$ norm. Based on our theorem, we propose a novel
regularization method termed norm decay, which can effectively reduce the norms
of convolutional layers and fully-connected layers. Experiments show that
norm-regularization methods, including norm decay, weight decay, and singular
value clipping, can improve generalization of CNNs. However, they can slightly
hurt adversarial robustness. Observing this unexpected phenomenon, we compute
the norms of layers in the CNNs trained with three different adversarial
training frameworks and surprisingly find that adversarially robust CNNs have
comparable or even larger layer norms than their non-adversarially robust
counterparts. Furthermore, we prove that under a mild assumption, adversarially
robust classifiers can be achieved using neural networks, and an adversarially
robust neural network can have an arbitrarily large Lipschitz constant. For
this reason, enforcing small norms on CNN layers may be neither necessary nor
effective in achieving adversarial robustness. The code is available at
https://github.com/youweiliang/norm_robustness.
