  3D multi-object tracking is a crucial component in the perception system of
autonomous driving vehicles. Tracking all dynamic objects around the vehicle is
essential for tasks such as obstacle avoidance and path planning. Autonomous
vehicles are usually equipped with different sensor modalities to improve
accuracy and reliability. While sensor fusion has been widely used in object
detection networks in recent years, most existing multi-object tracking
algorithms either rely on a single input modality, or do not fully exploit the
information provided by multiple sensing modalities. In this work, we propose
an end-to-end network for joint object detection and tracking based on radar
and camera sensor fusion. Our proposed method uses a center-based radar-camera
fusion algorithm for object detection and utilizes a greedy algorithm for
object association. The proposed greedy algorithm uses the depth, velocity and
2D displacement of the detected objects to associate them through time. This
makes our tracking algorithm very robust to occluded and overlapping objects,
as the depth and velocity information can help the network in distinguishing
them. We evaluate our method on the challenging nuScenes dataset, where it
achieves 20.0 AMOTA and outperforms all vision-based 3D tracking methods in the
benchmark, as well as the baseline LiDAR-based method. Our method is online
with a runtime of 35ms per image, making it very suitable for autonomous
driving applications.
