  Scaling object taxonomies is one of the important steps toward a robust
real-world deployment of recognition systems. We have faced remarkable progress
in images since the introduction of the LVIS benchmark. To continue this
success in videos, a new video benchmark, TAO, was recently presented. Given
the recent encouraging results from both detection and tracking communities, we
are interested in marrying those two advances and building a strong large
vocabulary video tracker. However, supervisions in LVIS and TAO are inherently
sparse or even missing, posing two new challenges for training the large
vocabulary trackers. First, no tracking supervisions are in LVIS, which leads
to inconsistent learning of detection (with LVIS and TAO) and tracking (only
with TAO). Second, the detection supervisions in TAO are partial, which results
in catastrophic forgetting of absent LVIS categories during video fine-tuning.
To resolve these challenges, we present a simple but effective learning
framework that takes full advantage of all available training data to learn
detection and tracking while not losing any LVIS categories to recognize. With
this new learning scheme, we show that consistent improvements of various large
vocabulary trackers are capable, setting strong baseline results on the
challenging TAO benchmarks.
