  3D multi-object tracking aims to uniquely and consistently identify all
mobile entities through time. Despite the rich spatiotemporal information
available in this setting, current 3D tracking methods primarily rely on
abstracted information and limited history, e.g. single-frame object bounding
boxes. In this work, we develop a holistic representation of traffic scenes
that leverages both spatial and temporal information of the actors in the
scene. Specifically, we reformulate tracking as a spatiotemporal problem by
representing tracked objects as sequences of time-stamped points and bounding
boxes over a long temporal history. At each timestamp, we improve the location
and motion estimates of our tracked objects through learned refinement over the
full sequence of object history. By considering time and space jointly, our
representation naturally encodes fundamental physical priors such as object
permanence and consistency across time. Our spatiotemporal tracking framework
achieves state-of-the-art performance on the Waymo and nuScenes benchmarks.
