  Gaussian process regression is widely used because of its ability to provide
well-calibrated uncertainty estimates and handle small or sparse datasets.
However, it struggles with high-dimensional data. One possible way to scale
this technique to higher dimensions is to leverage the implicit low-dimensional
manifold upon which the data actually lies, as postulated by the manifold
hypothesis. Prior work ordinarily requires the manifold structure to be
explicitly provided though, i.e. given by a mesh or be known to be one of the
well-known manifolds like the sphere. In contrast, in this paper we propose a
Gaussian process regression technique capable of inferring implicit structure
directly from data (labeled and unlabeled) in a fully differentiable way. For
the resulting model, we discuss its convergence to the Mat\'ern Gaussian
process on the assumed manifold. Our technique scales up to hundreds of
thousands of data points, and may improve the predictive performance and
calibration of the standard Gaussian process regression in high-dimensional
settings.
