  While separately leveraging monocular 3D object detection and 2D multi-object
tracking can be straightforwardly applied to sequence images in a
frame-by-frame fashion, stand-alone tracker cuts off the transmission of the
uncertainty from the 3D detector to tracking while cannot pass tracking error
differentials back to the 3D detector. In this work, we propose jointly
training 3D detection and 3D tracking from only monocular videos in an
end-to-end manner. The key component is a novel spatial-temporal information
flow module that aggregates geometric and appearance features to predict robust
similarity scores across all objects in current and past frames. Specifically,
we leverage the attention mechanism of the transformer, in which self-attention
aggregates the spatial information in a specific frame, and cross-attention
exploits relation and affinities of all objects in the temporal domain of
sequence frames. The affinities are then supervised to estimate the trajectory
and guide the flow of information between corresponding 3D objects. In
addition, we propose a temporal
  -consistency loss that explicitly involves 3D target motion modeling into the
learning, making the 3D trajectory smooth in the world coordinate system.
Time3D achieves 21.4\% AMOTA, 13.6\% AMOTP on the nuScenes 3D tracking
benchmark, surpassing all published competitors, and running at 38 FPS, while
Time3D achieves 31.2\% mAP, 39.4\% NDS on the nuScenes 3D detection benchmark.
