  Open-vocabulary Video Instance Segmentation (OpenVIS) can simultaneously
detect, segment, and track arbitrary object categories in a video, without
being constrained to categories seen during training. In this work, we propose
an OpenVIS framework called InstFormer that achieves powerful open vocabulary
capability through lightweight fine-tuning on a limited-category labeled
dataset. Specifically, InstFormer comes in three steps a) Open-world Mask
Proposal: we utilize a query-based transformer, which is encouraged to propose
all potential object instances, to obtain class-agnostic instance masks; b)
Open-vocabulary Instance Representation and Classification: we propose
InstCLIP, adapted from pre-trained CLIP with Instance Guidance Attention.
InstCLIP generates the instance token capable of representing each
open-vocabulary instance. These instance tokens not only enable open-vocabulary
classification for multiple instances with a single CLIP forward pass but have
also been proven effective for subsequent open-vocabulary instance tracking. c)
Rollout Association: we introduce a class-agnostic rollout tracker to predict
rollout tokens from the tracking tokens of previous frames to enable
open-vocabulary instance association across frames in the video. The
experimental results demonstrate the proposed InstFormer achieve
state-of-the-art capabilities on a comprehensive OpenVIS evaluation benchmark,
while also achieves competitive performance in fully supervised VIS task.
