  Single-View depth estimation using the CNNs trained from unlabelled videos
has shown significant promise. However, excellent results have mostly been
obtained in street-scene driving scenarios, and such methods often fail in
other settings, particularly indoor videos taken by handheld devices. In this
work, we establish that the complex ego-motions exhibited in handheld settings
are a critical obstacle for learning depth. Our fundamental analysis suggests
that the rotation behaves as noise during training, as opposed to the
translation (baseline) which provides supervision signals. To address the
challenge, we propose a data pre-processing method that rectifies training
images by removing their relative rotations for effective learning. The
significantly improved performance validates our motivation. Towards end-to-end
learning without requiring pre-processing, we propose an Auto-Rectify Network
with novel loss functions, which can automatically learn to rectify images
during training. Consequently, our results outperform the previous unsupervised
SOTA method by a large margin on the challenging NYUv2 dataset. We also
demonstrate the generalization of our trained model in ScanNet and Make3D, and
the universality of our proposed learning method on 7-Scenes and KITTI
datasets.
