  Recent advances in self-supervised learning (SSL) have largely closed the gap
with supervised ImageNet pretraining. Despite their success these methods have
been primarily applied to unlabeled ImageNet images, and show marginal gains
when trained on larger sets of uncurated images. We hypothesize that current
SSL methods perform best on iconic images, and struggle on complex scene images
with many objects. Analyzing contrastive SSL methods shows that they have poor
visual grounding and receive poor supervisory signal when trained on scene
images. We propose Contrastive Attention-Supervised Tuning(CAST) to overcome
these limitations. CAST uses unsupervised saliency maps to intelligently sample
crops, and to provide grounding supervision via a Grad-CAM attention loss.
Experiments on COCO show that CAST significantly improves the features learned
by SSL methods on scene images, and further experiments show that CAST-trained
models are more robust to changes in backgrounds.
