  3D point clouds are often perturbed by noise due to the inherent limitation
of acquisition equipments, which obstructs downstream tasks such as surface
reconstruction, rendering and so on. Previous works mostly infer the
displacement of noisy points from the underlying surface, which however are not
designated to recover the surface explicitly and may lead to sub-optimal
denoising results. To this end, we propose to learn the underlying manifold of
a noisy point cloud from differentiably subsampled points with trivial noise
perturbation and their embedded neighborhood feature, aiming to capture
intrinsic structures in point clouds. Specifically, we present an
autoencoder-like neural network. The encoder learns both local and non-local
feature representations of each point, and then samples points with low noise
via an adaptive differentiable pooling operation. Afterwards, the decoder
infers the underlying manifold by transforming each sampled point along with
the embedded feature of its neighborhood to a local surface centered around the
point. By resampling on the reconstructed manifold, we obtain a denoised point
cloud. Further, we design an unsupervised training loss, so that our network
can be trained in either an unsupervised or supervised fashion. Experiments
show that our method significantly outperforms state-of-the-art denoising
methods under both synthetic noise and real world noise. The code and data are
available at https://github.com/luost26/DMRDenoise
