  Large-scale Gaussian process inference has long faced practical challenges
due to time and space complexity that is superlinear in dataset size. While
sparse variational Gaussian process models are capable of learning from
large-scale data, standard strategies for sparsifying the model can prevent the
approximation of complex functions. In this work, we propose a novel
variational Gaussian process model that decouples the representation of mean
and covariance functions in reproducing kernel Hilbert space. We show that this
new parametrization generalizes previous models. Furthermore, it yields a
variational inference problem that can be solved by stochastic gradient ascent
with time and space complexity that is only linear in the number of mean
function parameters, regardless of the choice of kernels, likelihoods, and
inducing points. This strategy makes the adoption of large-scale expressive
Gaussian process models possible. We run several experiments on regression
tasks and show that this decoupled approach greatly outperforms previous sparse
variational Gaussian process inference procedures.
