  Deep Neural Networks are vulnerable to adversarial attacks. Among many
defense strategies, adversarial training with untargeted attacks is one of the
most effective methods. Theoretically, adversarial perturbation in untargeted
attacks can be added along arbitrary directions and the predicted labels of
untargeted attacks should be unpredictable. However, we find that the naturally
imbalanced inter-class semantic similarity makes those hard-class pairs become
virtual targets of each other. This study investigates the impact of such
closely-coupled classes on adversarial attacks and develops a self-paced
reweighting strategy in adversarial training accordingly. Specifically, we
propose to upweight hard-class pair losses in model optimization, which prompts
learning discriminative features from hard classes. We further incorporate a
term to quantify hard-class pair consistency in adversarial training, which
greatly boosts model robustness. Extensive experiments show that the proposed
adversarial training method achieves superior robustness performance over
state-of-the-art defenses against a wide range of adversarial attacks.
