  Self-driving cars must detect vehicles, pedestrians, and other traffic
participants accurately to operate safely. Small, far-away, or highly occluded
objects are particularly challenging because there is limited information in
the LiDAR point clouds for detecting them. To address this challenge, we
leverage valuable information from the past: in particular, data collected in
past traversals of the same scene. We posit that these past data, which are
typically discarded, provide rich contextual information for disambiguating the
above-mentioned challenging cases. To this end, we propose a novel, end-to-end
trainable Hindsight framework to extract this contextual information from past
traversals and store it in an easy-to-query data structure, which can then be
leveraged to aid future 3D object detection of the same scene. We show that
this framework is compatible with most modern 3D detection architectures and
can substantially improve their average precision on multiple autonomous
driving datasets, most notably by more than 300% on the challenging cases.
