  The general domain of video segmentation is currently fragmented into
different tasks spanning multiple benchmarks. Despite rapid progress in the
state-of-the-art, current methods are overwhelmingly task-specific and cannot
conceptually generalize to other tasks. Inspired by recent approaches with
multi-task capability, we propose TarViS: a novel, unified network architecture
that can be applied to any task that requires segmenting a set of arbitrarily
defined 'targets' in video. Our approach is flexible with respect to how tasks
define these targets, since it models the latter as abstract 'queries' which
are then used to predict pixel-precise target masks. A single TarViS model can
be trained jointly on a collection of datasets spanning different tasks, and
can hot-swap between tasks during inference without any task-specific
retraining. To demonstrate its effectiveness, we apply TarViS to four different
tasks, namely Video Instance Segmentation (VIS), Video Panoptic Segmentation
(VPS), Video Object Segmentation (VOS) and Point Exemplar-guided Tracking
(PET). Our unified, jointly trained model achieves state-of-the-art performance
on 5/7 benchmarks spanning these four tasks, and competitive performance on the
remaining two. Code and model weights are available at:
https://github.com/Ali2500/TarViS
