  We propose SfM-Net, a geometry-aware neural network for motion estimation in
videos that decomposes frame-to-frame pixel motion in terms of scene and object
depth, camera motion and 3D object rotations and translations. Given a sequence
of frames, SfM-Net predicts depth, segmentation, camera and rigid object
motions, converts those into a dense frame-to-frame motion field (optical
flow), differentiably warps frames in time to match pixels and back-propagates.
The model can be trained with various degrees of supervision: 1)
self-supervised by the re-projection photometric error (completely
unsupervised), 2) supervised by ego-motion (camera motion), or 3) supervised by
depth (e.g., as provided by RGBD sensors). SfM-Net extracts meaningful depth
estimates and successfully estimates frame-to-frame camera rotations and
translations. It often successfully segments the moving objects in the scene,
even though such supervision is never provided.
