  We introduce a novel monocular visual odometry (VO) system, NeRF-VO, that
integrates learning-based sparse visual odometry for low-latency camera
tracking and a neural radiance scene representation for fine-detailed dense
reconstruction and novel view synthesis. Our system initializes camera poses
using sparse visual odometry and obtains view-dependent dense geometry priors
from a monocular prediction network. We harmonize the scale of poses and dense
geometry, treating them as supervisory cues to train a neural implicit scene
representation. NeRF-VO demonstrates exceptional performance in both
photometric and geometric fidelity of the scene representation by jointly
optimizing a sliding window of keyframed poses and the underlying dense
geometry, which is accomplished through training the radiance field with volume
rendering. We surpass SOTA methods in pose estimation accuracy, novel view
synthesis fidelity, and dense reconstruction quality across a variety of
synthetic and real-world datasets while achieving a higher camera tracking
frequency and consuming less GPU memory.
