  Current multi-object tracking and segmentation (MOTS) methods follow the
tracking-by-detection paradigm and adopt convolutions for feature extraction.
However, as affected by the inherent receptive field, convolution based feature
extraction inevitably mixes up the foreground features and the background
features, resulting in ambiguities in the subsequent instance association. In
this paper, we propose a highly effective method for learning instance
embeddings based on segments by converting the compact image representation to
un-ordered 2D point cloud representation. Our method generates a new
tracking-by-points paradigm where discriminative instance embeddings are
learned from randomly selected points rather than images. Furthermore, multiple
informative data modalities are converted into point-wise representations to
enrich point-wise features. The resulting online MOTS framework, named
PointTrack, surpasses all the state-of-the-art methods including 3D tracking
methods by large margins (5.4% higher MOTSA and 18 times faster over
MOTSFusion) with the near real-time speed (22 FPS). Evaluations across three
datasets demonstrate both the effectiveness and efficiency of our method.
Moreover, based on the observation that current MOTS datasets lack crowded
scenes, we build a more challenging MOTS dataset named APOLLO MOTS with higher
instance density. Both APOLLO MOTS and our codes are publicly available at
https://github.com/detectRecog/PointTrack.
