  Deep neural networks are susceptible to adversarial attacks due to the
accumulation of perturbations in the feature level, and numerous works have
boosted model robustness by deactivating the non-robust feature activations
that cause model mispredictions. However, we claim that these malicious
activations still contain discriminative cues and that with recalibration, they
can capture additional useful information for correct model predictions. To
this end, we propose a novel, easy-to-plugin approach named Feature Separation
and Recalibration (FSR) that recalibrates the malicious, non-robust activations
for more robust feature maps through Separation and Recalibration. The
Separation part disentangles the input feature map into the robust feature with
activations that help the model make correct predictions and the non-robust
feature with activations that are responsible for model mispredictions upon
adversarial attack. The Recalibration part then adjusts the non-robust
activations to restore the potentially useful cues for model predictions.
Extensive experiments verify the superiority of FSR compared to traditional
deactivation techniques and demonstrate that it improves the robustness of
existing adversarial training methods by up to 8.57% with small computational
overhead. Codes are available at https://github.com/wkim97/FSR.
