  This paper delves into the challenges of achieving scalable and effective
multi-object modeling for semi-supervised Video Object Segmentation (VOS).
Previous VOS methods decode features with a single positive object, limiting
the learning of multi-object representation as they must match and segment each
target separately under multi-object scenarios. Additionally, earlier
techniques catered to specific application objectives and lacked the
flexibility to fulfill different speed-accuracy requirements. To address these
problems, we present two innovative approaches, Associating Objects with
Transformers (AOT) and Associating Objects with Scalable Transformers (AOST).
In pursuing effective multi-object modeling, AOT introduces the IDentification
(ID) mechanism to allocate each object a unique identity. This approach enables
the network to model the associations among all objects simultaneously, thus
facilitating the tracking and segmentation of objects in a single network pass.
To address the challenge of inflexible deployment, AOST further integrates
scalable long short-term transformers that incorporate scalable supervision and
layer-wise ID-based attention. This enables online architecture scalability in
VOS for the first time and overcomes ID embeddings' representation limitations.
Given the absence of a benchmark for VOS involving densely multi-object
annotations, we propose a challenging Video Object Segmentation in the Wild
(VOSW) benchmark to validate our approaches. We evaluated various AOT and AOST
variants using extensive experiments across VOSW and five commonly used VOS
benchmarks, including YouTube-VOS 2018 & 2019 Val, DAVIS-2017 Val & Test, and
DAVIS-2016. Our approaches surpass the state-of-the-art competitors and display
exceptional efficiency and scalability consistently across all six benchmarks.
Project page: https://github.com/yoxu515/aot-benchmark.
