  Current state-of-the-art open-vocabulary segmentation methods typically rely
on image-mask-text triplet annotations for supervision. However, acquiring such
detailed annotations is labour-intensive and poses scalability challenges in
complex real-world scenarios. While existing weakly-supervised approaches
leverage image-text pairs to reduce the expansive annotation cost, the lack of
mask supervision makes it difficult for the model to locate multiple instances
and accurately group pixels with similar semantics, significantly hampering
versatility and performance. In this paper, we introduce Unpair-Seg, a novel
weakly-supervised open-vocabulary segmentation framework that learns from
unpaired image-mask and image-text pairs, which can be independently and
efficiently collected. Unpair-Seg initially predicts a set of binary masks and
generates pseudo labels by identifying confident pairs of masks and text
entities. We then train a feature adapter to align region embeddings with text
embeddings based on these pseudo labels, achieving open-vocabulary
segmentation. However, the inherent noise in the mask-entity correspondence
poses a challenge to obtaining reliable pairs. To address this, we employ a
vision-language large model to re-caption the input images and extract precise
entities, and we design a multi-scale matching strategy to reduce noisy
mask-entity pairs. Our Unpair-Seg framework demonstrates impressive
performance, achieving 14.6\% and 19.5\% mIoU on the ADE-847 and PASCAL
Context-459 datasets, significantly narrowing the gap between fully-supervised
and weakly-supervised methods.
