  It is well-known that the performance of well-trained deep neural networks
may degrade significantly when they are applied to data with even slightly
shifted distributions. Recent studies have shown that introducing certain
perturbation on feature statistics (\eg, mean and standard deviation) during
training can enhance the cross-domain generalization ability. Existing methods
typically conduct such perturbation by utilizing the feature statistics within
a mini-batch, limiting their representation capability. Inspired by the domain
generalization objective, we introduce a novel Adversarial Style Augmentation
(ASA) method, which explores broader style spaces by generating more effective
statistics perturbation via adversarial training. Specifically, we first search
for the most sensitive direction and intensity for statistics perturbation by
maximizing the task loss. By updating the model against the adversarial
statistics perturbation during training, we allow the model to explore the
worst-case domain and hence improve its generalization performance. To
facilitate the application of ASA, we design a simple yet effective module,
namely AdvStyle, which instantiates the ASA method in a plug-and-play manner.
We justify the efficacy of AdvStyle on tasks of cross-domain classification and
instance retrieval. It achieves higher mean accuracy and lower performance
fluctuation. Especially, our method significantly outperforms its competitors
on the PACS dataset under the single source generalization setting, \eg,
boosting the classification accuracy from 61.2\% to 67.1\% with a ResNet50
backbone. Our code will be available at \url{https://github.com/YBZh/AdvStyle}.
