  Deep kernel learning (DKL) leverages the connection between Gaussian process
(GP) and neural networks (NN) to build an end-to-end, hybrid model. It combines
the capability of NN to learn rich representations under massive data and the
non-parametric property of GP to achieve automatic regularization that
incorporates a trade-off between model fit and model complexity. However, the
deterministic encoder may weaken the model regularization of the following GP
part, especially on small datasets, due to the free latent representation. We
therefore present a complete deep latent-variable kernel learning (DLVKL) model
wherein the latent variables perform stochastic encoding for regularized
representation. We further enhance the DLVKL from two aspects: (i) the
expressive variational posterior through neural stochastic differential
equation (NSDE) to improve the approximation quality, and (ii) the hybrid prior
taking knowledge from both the SDE prior and the posterior to arrive at a
flexible trade-off. Intensive experiments imply that the DLVKL-NSDE performs
similarly to the well calibrated GP on small datasets, and outperforms existing
deep GPs on large datasets.
