  Objection detection (OD) has been one of the most fundamental tasks in
computer vision. Recent developments in deep learning have pushed the
performance of image OD to new heights by learning-based, data-driven
approaches. On the other hand, video OD remains less explored, mostly due to
much more expensive data annotation needs. At the same time, multi-object
tracking (MOT) which requires reasoning about track identities and
spatio-temporal trajectories, shares similar spirits with video OD. However,
most MOT datasets are class-specific (e.g., person-annotated only), which
constrains a model's flexibility to perform tracking on other objects. We
propose TrIVD (Tracking and Image-Video Detection), the first framework that
unifies image OD, video OD, and MOT within one end-to-end model. To handle the
discrepancies and semantic overlaps of category labels across datasets, TrIVD
formulates detection/tracking as grounding and reasons about object categories
via visual-text alignments. The unified formulation enables cross-dataset,
multi-task training, and thus equips TrIVD with the ability to leverage
frame-level features, video-level spatio-temporal relations, as well as track
identity associations. With such joint training, we can now extend the
knowledge from OD data, that comes with much richer object category
annotations, to MOT and achieve zero-shot tracking capability. Experiments
demonstrate that multi-task co-trained TrIVD outperforms single-task baselines
across all image/video OD and MOT tasks. We further set the first baseline on
the new task of zero-shot tracking.
