  Efforts to improve the adversarial robustness of convolutional neural
networks have primarily focused on developing more effective adversarial
training methods. In contrast, little attention was devoted to analyzing the
role of architectural elements (such as topology, depth, and width) on
adversarial robustness. This paper seeks to bridge this gap and present a
holistic study on the impact of architectural design on adversarial robustness.
We focus on residual networks and consider architecture design at the block
level, i.e., topology, kernel size, activation, and normalization, as well as
at the network scaling level, i.e., depth and width of each block in the
network. In both cases, we first derive insights through systematic ablative
experiments. Then we design a robust residual block, dubbed RobustResBlock, and
a compound scaling rule, dubbed RobustScaling, to distribute depth and width at
the desired FLOP count. Finally, we combine RobustResBlock and RobustScaling
and present a portfolio of adversarially robust residual networks,
RobustResNets, spanning a broad spectrum of model capacities. Experimental
validation across multiple datasets and adversarial attacks demonstrate that
RobustResNets consistently outperform both the standard WRNs and other existing
robust architectures, achieving state-of-the-art AutoAttack robust accuracy of
61.1% without additional data and 63.7% with 500K external data while being
$2\times$ more compact in terms of parameters. Code is available at \url{
https://github.com/zhichao-lu/robust-residual-network}
