  Despite the remarkable success achieved by deep learning algorithms in
various domains, such as computer vision, they remain vulnerable to adversarial
perturbations. Adversarial Training (AT) stands out as one of the most
effective solutions to address this issue; however, single-step AT can lead to
Catastrophic Overfitting (CO). This scenario occurs when the adversarially
trained network suddenly loses robustness against multi-step attacks like
Projected Gradient Descent (PGD). Although several approaches have been
proposed to address this problem in Convolutional Neural Networks (CNNs), we
found out that they do not perform well when applied to Vision Transformers
(ViTs). In this paper, we propose Blacksmith, a novel training strategy to
overcome the CO problem, specifically in ViTs. Our approach utilizes either of
PGD-2 or Fast Gradient Sign Method (FGSM) randomly in a mini-batch during the
adversarial training of the neural network. This will increase the diversity of
our training attacks, which could potentially mitigate the CO issue. To manage
the increased training time resulting from this combination, we craft the PGD-2
attack based on only the first half of the layers, while FGSM is applied
end-to-end. Through our experiments, we demonstrate that our novel method
effectively prevents CO, achieves PGD-2 level performance, and outperforms
other existing techniques including N-FGSM, which is the state-of-the-art
method in fast training for CNNs.
