  Vision-language models trained on large, randomly collected data had
significant impact in many areas since they appeared. But as they show great
performance in various fields, such as image-text-retrieval, their inner
workings are still not fully understood. The current work analyses the true
zero-shot capabilities of those models. We start from the analysis of the
training corpus assessing to what extent (and which of) the test classes are
really zero-shot and how this correlates with individual classes performance.
We follow up with the analysis of the attribute-based zero-shot learning
capabilities of these models, evaluating how well this classical zero-shot
notion emerges from large-scale webly supervision. We leverage the recently
released LAION400M data corpus as well as the publicly available pretrained
models of CLIP, OpenCLIP, and FLAVA, evaluating the attribute-based zero-shot
capabilities on CUB and AWA2 benchmarks. Our analysis shows that: (i) most of
the classes in popular zero-shot benchmarks are observed (a lot) during
pre-training; (ii) zero-shot performance mainly comes out of models' capability
of recognizing class labels, whenever they are present in the text, and a
significantly lower performing capability of attribute-based zeroshot learning
is only observed when class labels are not used; (iii) the number of the
attributes used can have a significant effect on performance, and can easily
cause a significant performance decrease.
