  Gaussian process training decomposes into inference of the (approximate)
posterior and learning of the hyperparameters. For non-Gaussian (non-conjugate)
likelihoods, two common choices for approximate inference are Expectation
Propagation (EP) and Variational Inference (VI), which have complementary
strengths and weaknesses. While VI's lower bound to the marginal likelihood is
a suitable objective for inferring the approximate posterior, it does not
automatically imply it is a good learning objective for hyperparameter
optimization. We design a hybrid training procedure where the inference
leverages conjugate-computation VI and the learning uses an EP-like marginal
likelihood approximation. We empirically demonstrate on binary classification
that this provides a good learning objective and generalizes better.
