  Discriminative self-supervised learning allows training models on any random
group of internet images, and possibly recover salient information that helps
differentiate between the images. Applied to ImageNet, this leads to object
centric features that perform on par with supervised features on most
object-centric downstream tasks. In this work, we question if using this
ability, we can learn any salient and more representative information present
in diverse unbounded set of images from across the globe. To do so, we train
models on billions of random images without any data pre-processing or prior
assumptions about what we want the model to learn. We scale our model size to
dense 10 billion parameters to avoid underfitting on a large data size. We
extensively study and validate our model performance on over 50 benchmarks
including fairness, robustness to distribution shift, geographical diversity,
fine grained recognition, image copy detection and many image classification
datasets. The resulting model, not only captures well semantic information, it
also captures information about artistic style and learns salient information
such as geolocations and multilingual word embeddings based on visual content
only. More importantly, we discover that such model is more robust, more fair,
less harmful and less biased than supervised models or models trained on object
centric datasets such as ImageNet.
