  Modern multiple object tracking (MOT) systems usually follow the
\emph{tracking-by-detection} paradigm. It has 1) a detection model for target
localization and 2) an appearance embedding model for data association. Having
the two models separately executed might lead to efficiency problems, as the
running time is simply a sum of the two steps without investigating potential
structures that can be shared between them. Existing research efforts on
real-time MOT usually focus on the association step, so they are essentially
real-time association methods but not real-time MOT system. In this paper, we
propose an MOT system that allows target detection and appearance embedding to
be learned in a shared model. Specifically, we incorporate the appearance
embedding model into a single-shot detector, such that the model can
simultaneously output detections and the corresponding embeddings. We further
propose a simple and fast association method that works in conjunction with the
joint model. In both components the computation cost is significantly reduced
compared with former MOT systems, resulting in a neat and fast baseline for
future follow-ups on real-time MOT algorithm design. To our knowledge, this
work reports the first (near) real-time MOT system, with a running speed of 22
to 40 FPS depending on the input resolution. Meanwhile, its tracking accuracy
is comparable to the state-of-the-art trackers embodying separate detection and
embedding (SDE) learning ($64.4\%$ MOTA \vs $66.1\%$ MOTA on MOT-16 challenge).
Code and models are available at
\url{https://github.com/Zhongdao/Towards-Realtime-MOT}.
