  Transfer learning is a widely-used paradigm in deep learning, where models
pre-trained on standard datasets can be efficiently adapted to downstream
tasks. Typically, better pre-trained models yield better transfer results,
suggesting that initial accuracy is a key aspect of transfer learning
performance. In this work, we identify another such aspect: we find that
adversarially robust models, while less accurate, often perform better than
their standard-trained counterparts when used for transfer learning.
Specifically, we focus on adversarially robust ImageNet classifiers, and show
that they yield improved accuracy on a standard suite of downstream
classification tasks. Further analysis uncovers more differences between robust
and standard models in the context of transfer learning. Our results are
consistent with (and in fact, add to) recent hypotheses stating that robustness
leads to improved feature representations. Our code and models are available at
https://github.com/Microsoft/robust-models-transfer .
