  A commonly accepted hypothesis is that models with higher accuracy on
Imagenet perform better on other downstream tasks, leading to much research
dedicated to optimizing Imagenet accuracy. Recently this hypothesis has been
challenged by evidence showing that self-supervised models transfer better than
their supervised counterparts, despite their inferior Imagenet accuracy. This
calls for identifying the additional factors, on top of Imagenet accuracy, that
make models transferable. In this work we show that high diversity of the
features learnt by the model promotes transferability jointly with Imagenet
accuracy. Encouraged by the recent transferability results of self-supervised
models, we propose a method that combines self-supervised and supervised
pretraining to generate models with both high diversity and high accuracy, and
as a result high transferability. We demonstrate our results on several
architectures and multiple downstream tasks, including both single-label and
multi-label classification.
