  We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for
monocular Visual Odometry (VO). DPVO uses a novel recurrent network
architecture designed for tracking image patches across time. Recent approaches
to VO have significantly improved the state-of-the-art accuracy by using deep
networks to predict dense flow between video frames. However, using dense flow
incurs a large computational cost, making these previous methods impractical
for many use cases. Despite this, it has been assumed that dense flow is
important as it provides additional redundancy against incorrect matches. DPVO
disproves this assumption, showing that it is possible to get the best accuracy
and efficiency by exploiting the advantages of sparse patch-based matching over
dense flow. DPVO introduces a novel recurrent update operator for patch based
correspondence coupled with differentiable bundle adjustment. On Standard
benchmarks, DPVO outperforms all prior work, including the learning-based
state-of-the-art VO-system (DROID) using a third of the memory while running 3x
faster on average. Code is available at https://github.com/princeton-vl/DPVO
