  This paper presents a variational Bayesian kernel selection (VBKS) algorithm
for sparse Gaussian process regression (SGPR) models. In contrast to existing
GP kernel selection algorithms that aim to select only one kernel with the
highest model evidence, our proposed VBKS algorithm considers the kernel as a
random variable and learns its belief from data such that the uncertainty of
the kernel can be interpreted and exploited to avoid overconfident GP
predictions. To achieve this, we represent the probabilistic kernel as an
additional variational variable in a variational inference (VI) framework for
SGPR models where its posterior belief is learned together with that of the
other variational variables (i.e., inducing variables and kernel
hyperparameters). In particular, we transform the discrete kernel belief into a
continuous parametric distribution via reparameterization in order to apply VI.
Though it is computationally challenging to jointly optimize a large number of
hyperparameters due to many kernels being evaluated simultaneously by our VBKS
algorithm, we show that the variational lower bound of the log-marginal
likelihood can be decomposed into an additive form such that each additive term
depends only on a disjoint subset of the variational variables and can thus be
optimized independently. Stochastic optimization is then used to maximize the
variational lower bound by iteratively improving the variational approximation
of the exact posterior belief via stochastic gradient ascent, which incurs
constant time per iteration and hence scales to big data. We empirically
evaluate the performance of our VBKS algorithm on synthetic and massive
real-world datasets.
