  This work aims to leverage pre-trained foundation models, such as contrastive
language-image pre-training (CLIP) and segment anything model (SAM), to address
weakly supervised semantic segmentation (WSSS) using image-level labels. To
this end, we propose a coarse-to-fine framework based on CLIP and SAM for
generating high-quality segmentation seeds. Specifically, we construct an image
classification task and a seed segmentation task, which are jointly performed
by CLIP with frozen weights and two sets of learnable task-specific prompts. A
SAM-based seeding (SAMS) module is designed and applied to each task to produce
either coarse or fine seed maps. Moreover, we design a multi-label contrastive
loss supervised by image-level labels and a CAM activation loss supervised by
the generated coarse seed map. These losses are used to learn the prompts,
which are the only parts need to be learned in our framework. Once the prompts
are learned, we input each image along with the learned segmentation-specific
prompts into CLIP and the SAMS module to produce high-quality segmentation
seeds. These seeds serve as pseudo labels to train an off-the-shelf
segmentation network like other two-stage WSSS methods. Experiments show that
our method achieves the state-of-the-art performance on PASCAL VOC 2012 and
competitive results on MS COCO 2014. Code is available at
https://github.com/HAL-42/FMA-WSSS.git.
