  Adversarial training and its many variants substantially improve deep network
robustness, yet at the cost of compromising standard accuracy. Moreover, the
training process is heavy and hence it becomes impractical to thoroughly
explore the trade-off between accuracy and robustness. This paper asks this new
question: how to quickly calibrate a trained model in-situ, to examine the
achievable trade-offs between its standard and robust accuracies, without
(re-)training it many times? Our proposed framework, Once-for-all Adversarial
Training (OAT), is built on an innovative model-conditional training framework,
with a controlling hyper-parameter as the input. The trained model could be
adjusted among different standard and robust accuracies "for free" at testing
time. As an important knob, we exploit dual batch normalization to separate
standard and adversarial feature statistics, so that they can be learned in one
model without degrading performance. We further extend OAT to a Once-for-all
Adversarial Training and Slimming (OATS) framework, that allows for the joint
trade-off among accuracy, robustness and runtime efficiency. Experiments show
that, without any re-training nor ensembling, OAT/OATS achieve similar or even
superior performance compared to dedicatedly trained models at various
configurations. Our codes and pretrained models are available at:
https://github.com/VITA-Group/Once-for-All-Adversarial-Training.
