  Contrastive Language-Image Pretraining (CLIP) stands out as a prominent
method for image representation learning. Various neural architectures,
spanning Transformer-based models like Vision Transformers (ViTs) to
Convolutional Networks (ConvNets) like ResNets, are trained with CLIP and serve
as universal backbones across diverse vision tasks. Despite utilizing the same
data and training objectives, the effectiveness of representations learned by
these architectures raises a critical question. Our investigation explores the
differences in CLIP performance among these backbone architectures, revealing
significant disparities in their classifications. Notably, normalizing these
representations results in substantial performance variations. Our findings
showcase a remarkable possible synergy between backbone predictions that could
reach an improvement of over 20% through informed selection of the appropriate
backbone. Moreover, we propose a simple, yet effective approach to combine
predictions from multiple backbones, leading to a notable performance boost of
up to 6.34\%. We will release the code for reproducing the results.
