  Long-Tailed 3D Object Detection (LT3D) addresses the problem of accurately
detecting objects from both common and rare classes. Contemporary multi-modal
detectors achieve low AP on rare-classes (e.g., CMT only achieves 9.4 AP on
stroller), presumably because training detectors end-to-end with significant
class imbalance is challenging. To address this limitation, we delve into a
simple late-fusion framework that ensembles independently trained uni-modal
LiDAR and RGB detectors. Importantly, such a late-fusion framework allows us to
leverage large-scale uni-modal datasets (with more examples for rare classes)
to train better uni-modal RGB detectors, unlike prevailing multimodal detectors
that require paired multi-modal training data. Notably, our approach
significantly improves rare-class detection by 7.2% over prior work. Further,
we examine three critical components of our simple late-fusion approach from
first principles and investigate whether to train 2D or 3D RGB detectors,
whether to match RGB and LiDAR detections in 3D or the projected 2D image plane
for fusion, and how to fuse matched detections. Extensive experiments reveal
that 2D RGB detectors achieve better recognition accuracy for rare classes than
3D RGB detectors and matching on the 2D image plane mitigates depth estimation
errors. Our late-fusion approach achieves 51.4 mAP on the established nuScenes
LT3D benchmark, improving over prior work by 5.9 mAP!
