  We investigate learning of the differential geometric structure of a data
manifold embedded in a high-dimensional Euclidean space. We first analyze
kernel-based algorithms and show that under the usual regularizations,
non-probabilistic methods cannot recover the differential geometric structure,
but instead find mostly linear manifolds or spaces equipped with teleports. To
properly learn the differential geometric structure, non-probabilistic methods
must apply regularizations that enforce large gradients, which go against
common wisdom. We repeat the analysis for probabilistic methods and find that
under reasonable priors, the geometric structure can be recovered. Fully
exploiting the recovered structure, however, requires the development of
stochastic extensions to classic Riemannian geometry. We take early steps in
that regard. Finally, we partly extend the analysis to modern models based on
neural networks, thereby highlighting geometric and probabilistic shortcomings
of current deep generative models.
