  Robots navigating autonomously need to perceive and track the motion of
objects and other agents in its surroundings. This information enables planning
and executing robust and safe trajectories. To facilitate these processes, the
motion should be perceived in 3D Cartesian space. However, most recent
multi-object tracking (MOT) research has focused on tracking people and moving
objects in 2D RGB video sequences. In this work we present JRMOT, a novel 3D
MOT system that integrates information from RGB images and 3D point clouds to
achieve real-time, state-of-the-art tracking performance. Our system is built
with recent neural networks for re-identification, 2D and 3D detection and
track description, combined into a joint probabilistic data-association
framework within a multi-modal recursive Kalman architecture. As part of our
work, we release the JRDB dataset, a novel large scale 2D+3D dataset and
benchmark, annotated with over 2 million boxes and 3500 time consistent 2D+3D
trajectories across 54 indoor and outdoor scenes. JRDB contains over 60 minutes
of data including 360 degree cylindrical RGB video and 3D pointclouds in social
settings that we use to develop, train and evaluate JRMOT. The presented 3D MOT
system demonstrates state-of-the-art performance against competing methods on
the popular 2D tracking KITTI benchmark and serves as first 3D tracking
solution for our benchmark. Real-robot tests on our social robot JackRabbot
indicate that the system is capable of tracking multiple pedestrians fast and
reliably. We provide the ROS code of our tracker at
https://sites.google.com/view/jrmot.
