  Both self-supervised depth estimation and Structure-from-Motion (SfM) recover
scene depth from RGB videos. Despite sharing a similar objective, the two
approaches are disconnected. Prior works of self-supervision backpropagate
losses defined within immediate neighboring frames. Instead of
learning-through-loss, this work proposes an alternative scheme by performing
local SfM. First, with calibrated RGB or RGB-D images, we employ a depth and
correspondence estimator to infer depthmaps and pair-wise correspondence maps.
Then, a novel bundle-RANSAC-adjustment algorithm jointly optimizes camera poses
and one depth adjustment for each depthmap. Finally, we fix camera poses and
employ a NeRF, however, without a neural network, for dense triangulation and
geometric verification. Poses, depth adjustments, and triangulated sparse
depths are our outputs. For the first time, we show self-supervision within $5$
frames already benefits SoTA supervised depth and correspondence models. The
project page is held in the link (https://shngjz.github.io/SSfM.github.io/).
