  This paper introduces a network architecture to solve the
structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA),
which explicitly enforces multi-view geometry constraints in the form of
feature-metric error. The whole pipeline is differentiable so that the network
can learn suitable features that make the BA problem more tractable.
Furthermore, this work introduces a novel depth parameterization to recover
dense per-pixel depth. The network first generates several basis depth maps
according to the input image and optimizes the final depth as a linear
combination of these basis depth maps via feature-metric BA. The basis depth
maps generator is also learned via end-to-end training. The whole system nicely
combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and
deep learning (i.e. feature learning and basis depth maps learning) to address
the challenging dense SfM problem. Experiments on large scale real data prove
the success of the proposed method.
