  Adversarial training (AT) is one of the most effective ways for improving the
robustness of deep convolution neural networks (CNNs). Just like common network
training, the effectiveness of AT relies on the design of basic network
components. In this paper, we conduct an in-depth study on the role of the
basic ReLU activation component in AT for robust CNNs. We find that the
spatially-shared and input-independent properties of ReLU activation make CNNs
less robust to white-box adversarial attacks with either standard or
adversarial training. To address this problem, we extend ReLU to a novel Sparta
activation function (Spatially attentive and Adversarially Robust Activation),
which enables CNNs to achieve both higher robustness, i.e., lower error rate on
adversarial examples, and higher accuracy, i.e., lower error rate on clean
examples, than the existing state-of-the-art (SOTA) activation functions. We
further study the relationship between Sparta and the SOTA activation
functions, providing more insights about the advantages of our method. With
comprehensive experiments, we also find that the proposed method exhibits
superior cross-CNN and cross-dataset transferability. For the former, the
adversarially trained Sparta function for one CNN (e.g., ResNet-18) can be
fixed and directly used to train another adversarially robust CNN (e.g.,
ResNet-34). For the latter, the Sparta function trained on one dataset (e.g.,
CIFAR-10) can be employed to train adversarially robust CNNs on another dataset
(e.g., SVHN). In both cases, Sparta leads to CNNs with higher robustness than
the vanilla ReLU, verifying the flexibility and versatility of the proposed
method.
