  Multi-view camera-based 3D detection is a challenging problem in computer
vision. Recent works leverage a pretrained LiDAR detection model to transfer
knowledge to a camera-based student network. However, we argue that there is a
major domain gap between the LiDAR BEV features and the camera-based BEV
features, as they have different characteristics and are derived from different
sources. In this paper, we propose Geometry Enhanced Masked Image Modeling
(GeoMIM) to transfer the knowledge of the LiDAR model in a pretrain-finetune
paradigm for improving the multi-view camera-based 3D detection. GeoMIM is a
multi-camera vision transformer with Cross-View Attention (CVA) blocks that
uses LiDAR BEV features encoded by the pretrained BEV model as learning
targets. During pretraining, GeoMIM's decoder has a semantic branch completing
dense perspective-view features and the other geometry branch reconstructing
dense perspective-view depth maps. The depth branch is designed to be
camera-aware by inputting the camera's parameters for better transfer
capability. Extensive results demonstrate that GeoMIM outperforms existing
methods on nuScenes benchmark, achieving state-of-the-art performance for
camera-based 3D object detection and 3D segmentation. Code and pretrained
models are available at https://github.com/Sense-X/GeoMIM.
