  We propose a novel deep visual odometry (VO) method that considers global
information by selecting memory and refining poses. Existing learning-based
methods take the VO task as a pure tracking problem via recovering camera poses
from image snippets, leading to severe error accumulation. Global information
is crucial for alleviating accumulated errors. However, it is challenging to
effectively preserve such information for end-to-end systems. To deal with this
challenge, we design an adaptive memory module, which progressively and
adaptively saves the information from local to global in a neural analogue of
memory, enabling our system to process long-term dependency. Benefiting from
global information in the memory, previous results are further refined by an
additional refining module. With the guidance of previous outputs, we adopt a
spatial-temporal attention to select features for each view based on the
co-visibility in feature domain. Specifically, our architecture consisting of
Tracking, Remembering and Refining modules works beyond tracking. Experiments
on the KITTI and TUM-RGBD datasets demonstrate that our approach outperforms
state-of-the-art methods by large margins and produces competitive results
against classic approaches in regular scenes. Moreover, our model achieves
outstanding performance in challenging scenarios such as texture-less regions
and abrupt motions, where classic algorithms tend to fail.
