  Current methods for regularization in machine learning require quite specific
model assumptions (e.g. a kernel shape) that are not derived from prior
knowledge about the application, but must be imposed merely to make the method
work. We show in this paper that regularization can indeed be achieved by
assuming nothing but invariance principles (w.r.t. scaling, translation, and
rotation of input and output space) and the degree of differentiability of the
true function.
  Concretely, we derive a novel (non-Gaussian) stochastic process from the
above minimal assumptions, and we present a corresponding Bayesian inference
method for regression. The mean posterior turns out to be a polyharmonic
spline, and the posterior process is a mixture of t-processes.
  Compared with Gaussian process regression, the proposed method shows equal
performance and has the advantages of being (i) less arbitrary (no choice of
kernel) (ii) potentially faster (no kernel parameter optimization), and (iii)
having better extrapolation behavior.
  We believe that the proposed theory has central importance for the conceptual
foundations of regularization and machine learning and also has great potential
to enable practical advances in ML areas beyond regression.
