  Adversarial training has been empirically shown to be more prone to
overfitting than standard training. The exact underlying reasons still need to
be fully understood. In this paper, we identify one cause of overfitting
related to current practices of generating adversarial samples from
misclassified samples. To address this, we propose an alternative approach that
leverages the misclassified samples to mitigate the overfitting problem. We
show that our approach achieves better generalization while having comparable
robustness to state-of-the-art adversarial training methods on a wide range of
computer vision, natural language processing, and tabular tasks.
