  We introduce a new scalable variational Gaussian process approximation which
provides a high fidelity approximation while retaining general applicability.
We propose the harmonic kernel decomposition (HKD), which uses Fourier series
to decompose a kernel as a sum of orthogonal kernels. Our variational
approximation exploits this orthogonality to enable a large number of inducing
points at a low computational cost. We demonstrate that, on a range of
regression and classification problems, our approach can exploit input space
symmetries such as translations and reflections, and it significantly
outperforms standard variational methods in scalability and accuracy. Notably,
our approach achieves state-of-the-art results on CIFAR-10 among pure GP
models.
