  Current popular backbones in computer vision, such as Vision Transformers
(ViT) and ResNets are trained to perceive the world from 2D images. However, to
more effectively understand 3D structural priors in 2D backbones, we propose
Mask3D to leverage existing large-scale RGB-D data in a self-supervised
pre-training to embed these 3D priors into 2D learned feature representations.
In contrast to traditional 3D contrastive learning paradigms requiring 3D
reconstructions or multi-view correspondences, our approach is simple: we
formulate a pre-text reconstruction task by masking RGB and depth patches in
individual RGB-D frames. We demonstrate the Mask3D is particularly effective in
embedding 3D priors into the powerful 2D ViT backbone, enabling improved
representation learning for various scene understanding tasks, such as semantic
segmentation, instance segmentation and object detection. Experiments show that
Mask3D notably outperforms existing self-supervised 3D pre-training approaches
on ScanNet, NYUv2, and Cityscapes image understanding tasks, with an
improvement of +6.5% mIoU against the state-of-the-art Pri3D on ScanNet image
semantic segmentation.
