  Object tracking is central to robot perception and scene understanding.
Tracking-by-detection has long been a dominant paradigm for object tracking of
specific object categories. Recently, large-scale pre-trained models have shown
promising advances in detecting and segmenting objects and parts in 2D static
images in the wild. This begs the question: can we re-purpose these large-scale
pre-trained static image models for open-vocabulary video tracking? In this
paper, we re-purpose an open-vocabulary detector, segmenter, and dense optical
flow estimator, into a model that tracks and segments objects of any category
in 2D videos. Our method predicts object and part tracks with associated
language descriptions in monocular videos, rebuilding the pipeline of Tractor
with modern large pre-trained models for static image detection and
segmentation: we detect open-vocabulary object instances and propagate their
boxes from frame to frame using a flow-based motion model, refine the
propagated boxes with the box regression module of the visual detector, and
prompt an open-world segmenter with the refined box to segment the objects. We
decide the termination of an object track based on the objectness score of the
propagated boxes, as well as forward-backward optical flow consistency. We
re-identify objects across occlusions using deep feature matching. We show that
our model achieves strong performance on multiple established video object
segmentation and tracking benchmarks, and can produce reasonable tracks in
manipulation data. In particular, our model outperforms previous
state-of-the-art in UVO and BURST, benchmarks for open-world object tracking
and segmentation, despite never being explicitly trained for tracking. We hope
that our approach can serve as a simple and extensible framework for future
research.
