  Adversarial training is the industry standard for producing models that are
robust to small adversarial perturbations. However, machine learning
practitioners need models that are robust to other kinds of changes that occur
naturally, such as changes in the style or illumination of input images. Such
changes in input distribution have been effectively modeled as shifts in the
mean and variance of deep image features. We adapt adversarial training by
directly perturbing feature statistics, rather than image pixels, to produce
models that are robust to various unseen distributional shifts. We explore the
relationship between these perturbations and distributional shifts by
visualizing adversarial features. Our proposed method, Adversarial Batch
Normalization (AdvBN), is a single network layer that generates worst-case
feature perturbations during training. By fine-tuning neural networks on
adversarial feature distributions, we observe improved robustness of networks
to various unseen distributional shifts, including style variations and image
corruptions. In addition, we show that our proposed adversarial feature
perturbation can be complementary to existing image space data augmentation
methods, leading to improved performance. The source code and pre-trained
models are released at \url{https://github.com/azshue/AdvBN}.
