  The popular CLIP model displays impressive zero-shot capabilities thanks to
its seamless interaction with arbitrary text prompts. However, its lack of
spatial awareness makes it unsuitable for dense computer vision tasks, e.g.,
semantic segmentation, without an additional fine-tuning step that often uses
annotations and can potentially suppress its original open-vocabulary
properties. Meanwhile, self-supervised representation methods have demonstrated
good localization properties without human-made annotations nor explicit
supervision. In this work, we take the best of both worlds and propose an
open-vocabulary semantic segmentation method, which does not require any
annotations. We propose to locally improve dense MaskCLIP features, which are
computed with a simple modification of CLIP's last pooling layer, by
integrating localization priors extracted from self-supervised features. By
doing so, we greatly improve the performance of MaskCLIP and produce smooth
outputs. Moreover, we show that the used self-supervised feature properties can
directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a
single forward pass of CLIP and two light convolutional layers at inference, no
extra supervision nor extra memory and reaches state-of-the-art results on
challenging and fine-grained benchmarks such as COCO, Pascal Context,
Cityscapes and ADE20k. The code to reproduce our results is available at
https://github.com/wysoczanska/clip_dinoiser.
