  Existing referring understanding tasks tend to involve the detection of a
single text-referred object. In this paper, we propose a new and general
referring understanding task, termed referring multi-object tracking (RMOT).
Its core idea is to employ a language expression as a semantic cue to guide the
prediction of multi-object tracking. To the best of our knowledge, it is the
first work to achieve an arbitrary number of referent object predictions in
videos. To push forward RMOT, we construct one benchmark with scalable
expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18
videos with 818 expressions, and each expression in a video is annotated with
an average of 10.7 objects. Further, we develop a transformer-based
architecture TransRMOT to tackle the new task in an online manner, which
achieves impressive detection performance and outperforms other counterparts.
The dataset and code will be available at https://github.com/wudongming97/RMOT.
