  Adversarial examples derived from deliberately crafted perturbations on
visual inputs can easily harm decision process of deep neural networks. To
prevent potential threats, various adversarial training-based defense methods
have grown rapidly and become a de facto standard approach for robustness.
Despite recent competitive achievements, we observe that adversarial
vulnerability varies across targets and certain vulnerabilities remain
prevalent. Intriguingly, such peculiar phenomenon cannot be relieved even with
deeper architectures and advanced defense methods. To address this issue, in
this paper, we introduce a causal approach called Adversarial Double Machine
Learning (ADML), which allows us to quantify the degree of adversarial
vulnerability for network predictions and capture the effect of treatments on
outcome of interests. ADML can directly estimate causal parameter of
adversarial perturbations per se and mitigate negative effects that can
potentially damage robustness, bridging a causal perspective into the
adversarial vulnerability. Through extensive experiments on various CNN and
Transformer architectures, we corroborate that ADML improves adversarial
robustness with large margins and relieve the empirical observation.
