  Gaussian Process (GPs) models are a rich distribution over functions with
inductive biases controlled by a kernel function. Learning occurs through the
optimisation of kernel hyperparameters using the marginal likelihood as the
objective. This classical approach known as Type-II maximum likelihood (ML-II)
yields point estimates of the hyperparameters, and continues to be the default
method for training GPs. However, this approach risks underestimating
predictive uncertainty and is prone to overfitting especially when there are
many hyperparameters. Furthermore, gradient based optimisation makes ML-II
point estimates highly susceptible to the presence of local minima. This work
presents an alternative learning procedure where the hyperparameters of the
kernel function are marginalised using Nested Sampling (NS), a technique that
is well suited to sample from complex, multi-modal distributions. We focus on
regression tasks with the spectral mixture (SM) class of kernels and find that
a principled approach to quantifying model uncertainty leads to substantial
gains in predictive performance across a range of synthetic and benchmark data
sets. In this context, nested sampling is also found to offer a speed advantage
over Hamiltonian Monte Carlo (HMC), widely considered to be the gold-standard
in MCMC based inference.
