  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent
method for image representation learning. Various architectures, from vision
transformers (ViTs) to convolutional networks (ResNets) have been trained with
CLIP to serve as general solutions to diverse vision tasks. This paper explores
the differences across various CLIP-trained vision backbones. Despite using the
same data and training objective, we find that these architectures have notably
different representations, different classification performance across
datasets, and different robustness properties to certain types of image
perturbations. Our findings indicate a remarkable possible synergy across
backbones by leveraging their respective strengths. In principle,
classification accuracy could be improved by over 40 percentage with an
informed selection of the optimal backbone per test example.Using this insight,
we develop a straightforward yet powerful approach to adaptively ensemble
multiple backbones. The approach uses as few as one labeled example per class
to tune the adaptive combination of backbones. On a large collection of
datasets, the method achieves a remarkable increase in accuracy of up to 39.1%
over the best single backbone, well beyond traditional ensembles
