  Deep neural networks are incredibly vulnerable to crafted,
human-imperceptible adversarial perturbations. Although adversarial training
(AT) has proven to be an effective defense approach, we find that the
AT-trained models heavily rely on the input low-frequency content for judgment,
accounting for the low standard accuracy. To close the large gap between the
standard and robust accuracies during AT, we investigate the frequency
difference between clean and adversarial inputs, and propose a frequency
regularization (FR) to align the output difference in the spectral domain.
Besides, we find Stochastic Weight Averaging (SWA), by smoothing the kernels
over epochs, further improves the robustness. Among various defense schemes,
our method achieves the strongest robustness against attacks by PGD-20, C\&W
and Autoattack, on a WideResNet trained on CIFAR-10 without any extra data.
