  Tracking objects in three-dimensional space is critical for autonomous
driving. To ensure safety while driving, the tracker must be able to reliably
track objects across frames and accurately estimate their states such as
velocity and acceleration in the present. Existing works frequently focus on
the association task while either neglecting the model performance on state
estimation or deploying complex heuristics to predict the states. In this
paper, we propose STT, a Stateful Tracking model built with Transformers, that
can consistently track objects in the scenes while also predicting their states
accurately. STT consumes rich appearance, geometry, and motion signals through
long term history of detections and is jointly optimized for both data
association and state estimation tasks. Since the standard tracking metrics
like MOTA and MOTP do not capture the combined performance of the two tasks in
the wider spectrum of object states, we extend them with new metrics called
S-MOTA and MOTPS that address this limitation. STT achieves competitive
real-time performance on the Waymo Open Dataset.
