  A key contributor to recent progress in 3D detection from single images is
monocular depth estimation. Existing methods focus on how to leverage depth
explicitly, by generating pseudo-pointclouds or providing attention cues for
image features. More recent works leverage depth prediction as a pretraining
task and fine-tune the depth representation while training it for 3D detection.
However, the adaptation is insufficient and is limited in scale by manual
labels. In this work, we propose to further align depth representation with the
target domain in unsupervised fashions. Our methods leverage commonly available
LiDAR or RGB videos during training time to fine-tune the depth representation,
which leads to improved 3D detectors. Especially when using RGB videos, we show
that our two-stage training by first generating pseudo-depth labels is critical
because of the inconsistency in loss distribution between the two tasks. With
either type of reference data, our multi-task learning approach improves over
the state of the art on both KITTI and NuScenes, while matching the test-time
complexity of its single task sub-network.
