  We propose a method (TT-GP) for approximate inference in Gaussian Process
(GP) models. We build on previous scalable GP research including stochastic
variational inference based on inducing inputs, kernel interpolation, and
structure exploiting algebra. The key idea of our method is to use Tensor Train
decomposition for variational parameters, which allows us to train GPs with
billions of inducing inputs and achieve state-of-the-art results on several
benchmarks. Further, our approach allows for training kernels based on deep
neural networks without any modifications to the underlying GP model. A neural
network learns a multidimensional embedding for the data, which is used by the
GP to make the final prediction. We train GP and neural network parameters
end-to-end without pretraining, through maximization of GP marginal likelihood.
We show the efficiency of the proposed approach on several regression and
classification benchmark datasets including MNIST, CIFAR-10, and Airline.
