  Multi-sensor perception is crucial to ensure the reliability and accuracy in
autonomous driving system, while multi-object tracking (MOT) improves that by
tracing sequential movement of dynamic objects. Most current approaches for
multi-sensor multi-object tracking are either lack of reliability by tightly
relying on a single input source (e.g., center camera), or not accurate enough
by fusing the results from multiple sensors in post processing without fully
exploiting the inherent information. In this study, we design a generic
sensor-agnostic multi-modality MOT framework (mmMOT), where each modality
(i.e., sensors) is capable of performing its role independently to preserve
reliability, and further improving its accuracy through a novel multi-modality
fusion module. Our mmMOT can be trained in an end-to-end manner, enables joint
optimization for the base feature extractor of each modality and an adjacency
estimator for cross modality. Our mmMOT also makes the first attempt to encode
deep representation of point cloud in data association process in MOT. We
conduct extensive experiments to evaluate the effectiveness of the proposed
framework on the challenging KITTI benchmark and report state-of-the-art
performance. Code and models are available at
https://github.com/ZwwWayne/mmMOT.
