  This paper investigates how to realize better and more efficient embedding
learning to tackle the semi-supervised video object segmentation under
challenging multi-object scenarios. The state-of-the-art methods learn to
decode features with a single positive object and thus have to match and
segment each target separately under multi-object scenarios, consuming multiple
times computing resources. To solve the problem, we propose an Associating
Objects with Transformers (AOT) approach to match and decode multiple objects
uniformly. In detail, AOT employs an identification mechanism to associate
multiple targets into the same high-dimensional embedding space. Thus, we can
simultaneously process multiple objects' matching and segmentation decoding as
efficiently as processing a single object. For sufficiently modeling
multi-object association, a Long Short-Term Transformer is designed for
constructing hierarchical matching and propagation. We conduct extensive
experiments on both multi-object and single-object benchmarks to examine AOT
variant networks with different complexities. Particularly, our R50-AOT-L
outperforms all the state-of-the-art competitors on three popular benchmarks,
i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%),
while keeping more than $3\times$ faster multi-object run-time. Meanwhile, our
AOT-T can maintain real-time multi-object speed on the above benchmarks. Based
on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.
