  This paper focuses on training a robust RGB-D registration model without
ground-truth pose supervision. Existing methods usually adopt a pairwise
training strategy based on differentiable rendering, which enforces the
photometric and the geometric consistency between the two registered frames as
supervision. However, this frame-to-frame framework suffers from poor
multi-view consistency due to factors such as lighting changes, geometry
occlusion and reflective materials. In this paper, we present NeRF-UR, a novel
frame-to-model optimization framework for unsupervised RGB-D registration.
Instead of frame-to-frame consistency, we leverage the neural radiance field
(NeRF) as a global model of the scene and use the consistency between the input
and the NeRF-rerendered frames for pose optimization. This design can
significantly improve the robustness in scenarios with poor multi-view
consistency and provides better learning signal for the registration model.
Furthermore, to bootstrap the NeRF optimization, we create a synthetic dataset,
Sim-RGBD, through a photo-realistic simulator to warm up the registration
model. By first training the registration model on Sim-RGBD and later
unsupervisedly fine-tuning on real data, our framework enables distilling the
capability of feature extraction and registration from simulation to reality.
Our method outperforms the state-of-the-art counterparts on two popular indoor
RGB-D datasets, ScanNet and 3DMatch. Code and models will be released for paper
reproduction.
