  Adversarial attacks have been proven to be potential threats to Deep Neural
Networks (DNNs), and many methods are proposed to defend against adversarial
attacks. However, while enhancing the robustness, the clean accuracy will
decline to a certain extent, implying a trade-off existed between the accuracy
and robustness. In this paper, to meet the trade-off problem, we theoretically
explore the underlying reason for the difference of the filters' weight
distribution between standard-trained and robust-trained models and then argue
that this is an intrinsic property for static neural networks, thus they are
difficult to fundamentally improve the accuracy and adversarial robustness at
the same time. Based on this analysis, we propose a sample-wise dynamic network
architecture named Adversarial Weight-Varied Network (AW-Net), which focuses on
dealing with clean and adversarial examples with a "divide and rule" weight
strategy. The AW-Net adaptively adjusts the network's weights based on
regulation signals generated by an adversarial router, which is directly
influenced by the input sample. Benefiting from the dynamic network
architecture, clean and adversarial examples can be processed with different
network weights, which provides the potential to enhance both accuracy and
adversarial robustness. A series of experiments demonstrate that our AW-Net is
architecture-friendly to handle both clean and adversarial examples and can
achieve better trade-off performance than state-of-the-art robust models.
