  Generalized Referring Expression Segmentation (GRES) extends the scope of
classic RES to refer to multiple objects in one expression or identify the
empty targets absent in the image. GRES poses challenges in modeling the
complex spatial relationships of the instances in the image and identifying
non-existing referents. Multimodal Large Language Models (MLLMs) have recently
shown tremendous progress in these complicated vision-language tasks.
Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient
in understanding contexts with visual inputs. Among them, LISA, as a
representative, adopts a special [SEG] token to prompt a segmentation mask
decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing
solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot
correctly handle the cases where users might reference multiple subjects in a
singular prompt or provide descriptions incongruent with any image target. In
this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to
address this gap. Specifically, GSVA reuses the [SEG] token to prompt the
segmentation model towards supporting multiple mask references simultaneously
and innovatively learns to generate a [REJ] token to reject the null targets
explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue,
marking a notable enhancement and setting a new record on the GRES benchmark
gRefCOCO dataset. GSVA also proves effective across various classic referring
segmentation and comprehension tasks.
