  Vision-language (VL) pre-training has recently gained much attention for its
transferability and flexibility in novel concepts (e.g., cross-modality
transfer) across various visual tasks. However, VL-driven segmentation has been
under-explored, and the existing approaches still have the burden of acquiring
additional training images or even segmentation annotations to adapt a VL model
to downstream segmentation tasks. In this paper, we introduce a novel
image-free segmentation task where the goal is to perform semantic segmentation
given only a set of the target semantic categories, but without any
task-specific images and annotations. To tackle this challenging task, our
proposed method, coined IFSeg, generates VL-driven artificial
image-segmentation pairs and updates a pre-trained VL model to a segmentation
task. We construct this artificial training data by creating a 2D map of random
semantic categories and another map of their corresponding word tokens. Given
that a pre-trained VL model projects visual and text tokens into a common space
where tokens that share the semantics are located closely, this artificially
generated word map can replace the real image inputs for such a VL model.
Through an extensive set of experiments, our model not only establishes an
effective baseline for this novel task but also demonstrates strong
performances compared to existing methods that rely on stronger supervision,
such as task-specific images and segmentation masks. Code is available at
https://github.com/alinlab/ifseg.
