  Learning non-rigid registration in an end-to-end manner is challenging due to
the inherent high degrees of freedom and the lack of labeled training data. In
this paper, we resolve these two challenges simultaneously. First, we propose
to represent the non-rigid transformation with a point-wise combination of
several rigid transformations. This representation not only makes the solution
space well-constrained but also enables our method to be solved iteratively
with a recurrent framework, which greatly reduces the difficulty of learning.
Second, we introduce a differentiable loss function that measures the 3D shape
similarity on the projected multi-view 2D depth images so that our full
framework can be trained end-to-end without ground truth supervision. Extensive
experiments on several different datasets demonstrate that our proposed method
outperforms the previous state-of-the-art by a large margin. The source codes
are available at https://github.com/WanquanF/RMA-Net.
