  Deep normal estimators have made great strides on synthetic benchmarks.
Unfortunately, their performance dramatically drops on the real scan data since
they are supervised only on synthetic datasets. The point-wise annotation of
ground truth normals is vulnerable to inefficiency and inaccuracies, which
totally makes it impossible to build perfect real datasets for supervised deep
learning. To overcome the challenge, we propose a multi-sample consensus
paradigm for unsupervised normal estimation. The paradigm consists of
multi-candidate sampling, candidate rejection, and mode determination. The
latter two are driven by neighbor point consensus and candidate consensus
respectively. Two primary implementations of the paradigm, MSUNE and MSUNE-Net,
are proposed. MSUNE minimizes a candidate consensus loss in mode determination.
As a robust optimization method, it outperforms the cutting-edge supervised
deep learning methods on real data at the cost of longer runtime for sampling
enough candidate normals for each query point. MSUNE-Net, the first
unsupervised deep normal estimator as far as we know, significantly promotes
the multi-sample consensus further. It transfers the three online stages of
MSUNE to offline training. Thereby its inference time is 100 times faster.
Besides that, more accurate inference is achieved, since the candidates of
query points from similar patches can form a sufficiently large candidate set
implicitly in MSUNE-Net. Comprehensive experiments demonstrate that the two
proposed unsupervised methods are noticeably superior to some supervised deep
normal estimators on the most common synthetic dataset. More importantly, they
show better generalization ability and outperform all the SOTA conventional and
deep methods on three real datasets: NYUV2, KITTI, and a dataset from PCV [1].
