  All current non-rigid structure from motion (NRSfM) algorithms are limited
with respect to: (i) the number of images, and (ii) the type of shape
variability they can handle. This has hampered the practical utility of NRSfM
for many applications within vision. In this paper we propose a novel deep
neural network to recover camera poses and 3D points solely from an ensemble of
2D image coordinates. The proposed neural network is mathematically
interpretable as a multi-layer block sparse dictionary learning problem, and
can handle problems of unprecedented scale and shape complexity. Extensive
experiments demonstrate the impressive performance of our approach where we
exhibit superior precision and robustness against all available
state-of-the-art works. The considerable model capacity of our approach affords
remarkable generalization to unseen data. We propose a quality measure (based
on the network weights) which circumvents the need for 3D ground-truth to
ascertain the confidence we have in the reconstruction. Once the network's
weights are estimated (for a non-rigid object) we show how our approach can
effectively recover 3D shape from a single image -- outperforming comparable
methods that rely on direct 3D supervision.
