  Variational inference techniques based on inducing variables provide an
elegant framework for scalable posterior estimation in Gaussian process (GP)
models. Besides enabling scalability, one of their main advantages over sparse
approximations using direct marginal likelihood maximization is that they
provide a robust alternative for point estimation of the inducing inputs, i.e.
the location of the inducing variables. In this work we challenge the common
wisdom that optimizing the inducing inputs in the variational framework yields
optimal performance. We show that, by revisiting old model approximations such
as the fully-independent training conditionals endowed with powerful
sampling-based inference methods, treating both inducing locations and GP
hyper-parameters in a Bayesian way can improve performance significantly. Based
on stochastic gradient Hamiltonian Monte Carlo, we develop a fully Bayesian
approach to scalable GP and deep GP models, and demonstrate its
state-of-the-art performance through an extensive experimental campaign across
several regression and classification problems.
