  Approximate inference in Gaussian process (GP) models with non-conjugate
likelihoods gets entangled with the learning of the model hyperparameters. We
improve hyperparameter learning in GP models and focus on the interplay between
variational inference (VI) and the learning target. While VI's lower bound to
the marginal likelihood is a suitable objective for inferring the approximate
posterior, we show that a direct approximation of the marginal likelihood as in
Expectation Propagation (EP) is a better learning objective for hyperparameter
optimization. We design a hybrid training procedure to bring the best of both
worlds: it leverages conjugate-computation VI for inference and uses an EP-like
marginal likelihood approximation for hyperparameter learning. We compare VI,
EP, Laplace approximation, and our proposed training procedure and empirically
demonstrate the effectiveness of our proposal across a wide range of data sets.
