  Recently, FGSM adversarial training is found to be able to train a robust
model which is comparable to the one trained by PGD but an order of magnitude
faster. However, there is a failure mode called catastrophic overfitting (CO)
that the classifier loses its robustness suddenly during the training and
hardly recovers by itself. In this paper, we find CO is not only limited to
FGSM, but also happens in $\mbox{DF}^{\infty}$-1 adversarial training. Then, we
analyze the geometric properties for both FGSM and $\mbox{DF}^{\infty}$-1 and
find they have totally different decision boundaries after CO. For FGSM, a new
decision boundary is generated along the direction of perturbation and makes
the small perturbation more effective than the large one. While for
$\mbox{DF}^{\infty}$-1, there is no new decision boundary generated along the
direction of perturbation, instead the perturbation generated by
$\mbox{DF}^{\infty}$-1 becomes smaller after CO and thus loses its
effectiveness. We also experimentally analyze three hypotheses on potential
factors causing CO. And then based on the empirical analysis, we modify the
RS-FGSM by not projecting perturbation back to the $l_\infty$ ball. By this
small modification, we could achieve $47.56 \pm 0.37\% $ PGD-50-10 accuracy on
CIFAR10 with $\epsilon=8/255$ in contrast to $43.57 \pm 0.30\% $ by RS-FGSM and
also further extend the working range of $\epsilon$ from 8/255 to 11/255 on
CIFAR10 without CO occurring.
