  Learning robust and scalable visual representations from massive multi-view
video data remains a challenge in computer vision and autonomous driving.
Existing pre-training methods either rely on expensive supervised learning with
3D annotations, limiting the scalability, or focus on single-frame or monocular
inputs, neglecting the temporal information. We propose MIM4D, a novel
pre-training paradigm based on dual masked image modeling (MIM). MIM4D
leverages both spatial and temporal relations by training on masked multi-view
video inputs. It constructs pseudo-3D features using continuous scene flow and
projects them onto 2D plane for supervision. To address the lack of dense 3D
supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable
rendering to learn geometric representations. We demonstrate that MIM4D
achieves state-of-the-art performance on the nuScenes dataset for visual
representation learning in autonomous driving. It significantly improves
existing methods on multiple downstream tasks, including BEV segmentation (8.7%
IoU), 3D object detection (3.5% mAP), and HD map construction (1.4% mAP). Our
work offers a new choice for learning representation at scale in autonomous
driving. Code and models are released at https://github.com/hustvl/MIM4D
