  We propose a novel framework for training neural networks which is capable of
learning 3D information of non-rigid objects when only 2D annotations are
available as ground truths. Recently, there have been some approaches that
incorporate the problem setting of non-rigid structure-from-motion (NRSfM) into
deep learning to learn 3D structure reconstruction. The most important
difficulty of NRSfM is to estimate both the rotation and deformation at the
same time, and previous works handle this by regressing both of them. In this
paper, we resolve this difficulty by proposing a loss function wherein the
suitable rotation is automatically determined. Trained with the cost function
consisting of the reprojection error and the low-rank term of aligned shapes,
the network learns the 3D structures of such objects as human skeletons and
faces during the training, whereas the testing is done in a single-frame basis.
The proposed method can handle inputs with missing entries and experimental
results validate that the proposed framework shows superior reconstruction
performance to the state-of-the-art method on the Human 3.6M, 300-VW, and
SURREAL datasets, even though the underlying network structure is very simple.
