  Catastrophic overfitting (CO) in single-step adversarial training (AT)
results in abrupt drops in the adversarial test accuracy (even down to 0%). For
models trained with multi-step AT, it has been observed that the loss function
behaves locally linearly with respect to the input, this is however lost in
single-step AT. To address CO in single-step AT, several methods have been
proposed to enforce local linearity of the loss via regularization. However,
these regularization terms considerably slow down training due to Double
Backpropagation. Instead, in this work, we introduce a regularization term,
called ELLE, to mitigate CO effectively and efficiently in classical AT
evaluations, as well as some more difficult regimes, e.g., large adversarial
perturbations and long training schedules. Our regularization term can be
theoretically linked to curvature of the loss function and is computationally
cheaper than previous methods by avoiding Double Backpropagation. Our thorough
experimental validation demonstrates that our work does not suffer from CO,
even in challenging settings where previous works suffer from it. We also
notice that adapting our regularization parameter during training (ELLE-A)
greatly improves the performance, specially in large $\epsilon$ setups. Our
implementation is available in https://github.com/LIONS-EPFL/ELLE .
