  Predicting depth from a monocular video sequence is an important task for
autonomous driving. Although it has advanced considerably in the past few
years, recent methods based on convolutional neural networks (CNNs) discard
temporal coherence in the video sequence and estimate depth independently for
each frame, which often leads to undesired inconsistent results over time. To
address this problem, we propose to memorize temporal consistency in the video
sequence, and leverage it for the task of depth prediction. To this end, we
introduce a two-stream CNN with a flow-guided memory module, where each stream
encodes visual and temporal features, respectively. The memory module,
implemented using convolutional gated recurrent units (ConvGRUs), inputs visual
and temporal features sequentially together with optical flow tailored to our
task. It memorizes trajectories of individual features selectively and
propagates spatial information over time, enforcing a long-term temporal
consistency to prediction results. We evaluate our method on the KITTI
benchmark dataset in terms of depth prediction accuracy, temporal consistency
and runtime, and achieve a new state of the art. We also provide an extensive
experimental analysis, clearly demonstrating the effectiveness of our approach
to memorizing temporal consistency for depth prediction.
