  This paper is an attempt to bridge the conceptual gaps between researchers
working on the two widely used approaches based on positive definite kernels:
Bayesian learning or inference using Gaussian processes on the one side, and
frequentist kernel methods based on reproducing kernel Hilbert spaces on the
other. It is widely known in machine learning that these two formalisms are
closely related; for instance, the estimator of kernel ridge regression is
identical to the posterior mean of Gaussian process regression. However, they
have been studied and developed almost independently by two essentially
separate communities, and this makes it difficult to seamlessly transfer
results between them. Our aim is to overcome this potential difficulty. To this
end, we review several old and new results and concepts from either side, and
juxtapose algorithmic quantities from each framework to highlight close
similarities. We also provide discussions on subtle philosophical and
theoretical differences between the two approaches.
