  Tracking any given object(s) spatially and temporally is a common purpose in
Visual Object Tracking (VOT) and Video Object Segmentation (VOS). Joint
tracking and segmentation have been attempted in some studies but they often
lack full compatibility of both box and mask in initialization and prediction,
and mainly focus on single-object scenarios. To address these limitations, this
paper proposes a Multi-object Mask-box Integrated framework for unified
Tracking and Segmentation, dubbed MITS. Firstly, the unified identification
module is proposed to support both box and mask reference for initialization,
where detailed object information is inferred from boxes or directly retained
from masks. Additionally, a novel pinpoint box predictor is proposed for
accurate multi-object box prediction, facilitating target-oriented
representation learning. All target objects are processed simultaneously from
encoding to propagation and decoding, as a unified pipeline for VOT and VOS.
Experimental results show MITS achieves state-of-the-art performance on both
VOT and VOS benchmarks. Notably, MITS surpasses the best prior VOT competitor
by around 6% on the GOT-10k test set, and significantly improves the
performance of box initialization on VOS benchmarks. The code is available at
https://github.com/yoxu515/MITS.
