  We introduce a new interpretation of sparse variational approximations for
Gaussian processes using inducing points, which can lead to more scalable
algorithms than previous methods. It is based on decomposing a Gaussian process
as a sum of two independent processes: one spanned by a finite basis of
inducing points and the other capturing the remaining variation. We show that
this formulation recovers existing approximations and at the same time allows
to obtain tighter lower bounds on the marginal likelihood and new stochastic
variational inference algorithms. We demonstrate the efficiency of these
algorithms in several Gaussian process models ranging from standard regression
to multi-class classification using (deep) convolutional Gaussian processes and
report state-of-the-art results on CIFAR-10 among purely GP-based models.
