  Deep neural networks are vulnerable to adversarial samples. Adversarial
fine-tuning methods aim to enhance adversarial robustness through fine-tuning
the naturally pre-trained model in an adversarial training manner. However, we
identify that some latent features of adversarial samples are confused by
adversarial perturbation and lead to an unexpectedly increasing gap between
features in the last hidden layer of natural and adversarial samples. To
address this issue, we propose a disentanglement-based approach to explicitly
model and further remove the latent features that cause the feature gap.
Specifically, we introduce a feature disentangler to separate out the latent
features from the features of the adversarial samples, thereby boosting
robustness by eliminating the latent features. Besides, we align features in
the pre-trained model with features of adversarial samples in the fine-tuned
model, to further benefit from the features from natural samples without
confusion. Empirical evaluations on three benchmark datasets demonstrate that
our approach surpasses existing adversarial fine-tuning methods and adversarial
training baselines.
