  Deep Neural Network (DNN) are vulnerable to adversarial attacks. As a
countermeasure, adversarial training aims to achieve robustness based on the
min-max optimization problem and it has shown to be one of the most effective
defense strategies. However, in this work, we found that compared with natural
training, adversarial training fails to learn better feature representations
for either clean or adversarial samples, which can be one reason why
adversarial training tends to have severe overfitting issues and less satisfied
generalize performance. Specifically, we observe two major shortcomings of the
features learned by existing adversarial training methods:(1) low intra-class
feature similarity; and (2) conservative inter-classes feature variance. To
overcome these shortcomings, we introduce a new concept of adversarial training
graph (ATG) with which the proposed adversarial training with feature
separability (ATFS) enables to coherently boost the intra-class feature
similarity and increase inter-class feature variance. Through comprehensive
experiments, we demonstrate that the proposed ATFS framework significantly
improves both clean and robust performance.
