  Vision-language contrastive learning frameworks like CLIP enable learning
representations from natural language supervision, and provide strong zero-shot
classification capabilities. However, due to the nature of the supervisory
signal in these paradigms, they lack the ability to learn localized features,
leading to degraded performance on dense prediction tasks like segmentation and
detection. On the other hand, self-supervised learning methods have shown the
ability to learn granular representations, complementing the high-level
features in vision-language training. In this work, we present Harmony, a
framework that combines vision-language training with discriminative and
generative self-supervision to learn visual features that can be generalized
across vision downstream tasks. Our framework is specifically designed to work
on web-scraped data by not relying on negative examples and addressing the
one-to-one correspondence issue using soft CLIP targets generated by an EMA
model. We comprehensively evaluate Harmony across various vision downstream
tasks and find that it significantly outperforms the baseline CLIP and the
previously leading joint self and weakly-supervised methods, MaskCLIP and SLIP.
Specifically, when comparing against these methods, Harmony shows superior
performance in fine-tuning and zero-shot classification on ImageNet-1k,
semantic segmentation on ADE20K, and both object detection and instance
segmentation on MS-COCO, when pre-training a ViT-S/16 on CC3M. We also show
that Harmony outperforms other self-supervised learning methods like iBOT and
MAE across all tasks evaluated. On https://github.com/MohammedSB/Harmony our
code is publicly available.
