  Recently, semantic segmentation models trained with image-level text
supervision have shown promising results in challenging open-world scenarios.
However, these models still face difficulties in learning fine-grained semantic
alignment at the pixel level and predicting accurate object masks. To address
this issue, we propose MixReorg, a novel and straightforward pre-training
paradigm for semantic segmentation that enhances a model's ability to
reorganize patches mixed across images, exploring both local visual relevance
and global semantic coherence. Our approach involves generating fine-grained
patch-text pairs data by mixing image patches while preserving the
correspondence between patches and text. The model is then trained to minimize
the segmentation loss of the mixed images and the two contrastive losses of the
original and restored features. With MixReorg as a mask learner, conventional
text-supervised semantic segmentation models can achieve highly generalizable
pixel-semantic alignment ability, which is crucial for open-world segmentation.
After training with large-scale image-text data, MixReorg models can be applied
directly to segment visual objects of arbitrary categories, without the need
for further fine-tuning. Our proposed framework demonstrates strong performance
on popular zero-shot semantic segmentation benchmarks, outperforming GroupViT
by significant margins of 5.0%, 6.2%, 2.5%, and 3.4% mIoU on PASCAL VOC2012,
PASCAL Context, MS COCO, and ADE20K, respectively.
