  Vision language (VL) models like CLIP are robust to natural distribution
shifts, in part because CLIP learns on unstructured data using a technique
called caption supervision; the model inteprets image-linked texts as
ground-truth labels. In a carefully controlled comparison study, we show that
caption-supervised CNNs trained on a standard cross-entropy loss (with image
labels assigned by scanning captions for class names) can exhibit greater
distributional robustness than VL models trained on the same data. To
facilitate future experiments with high-accuracy caption-supervised models, we
introduce CaptionNet (https://github.com/penfever/CaptionNet/), which includes
a class-balanced, fully supervised dataset with over 50,000 new human-labeled
ImageNet-compliant samples which includes web-scraped captions. In a series of
experiments on CaptionNet, we show how the choice of loss function, data
filtration and supervision strategy enable robust computer vision. We also
provide the codebase necessary to reproduce our experiments at VL Hub
(https://github.com/penfever/vlhub/).
