  Deep kernel learning (DKL) and related techniques aim to combine the
representational power of neural networks with the reliable uncertainty
estimates of Gaussian processes. One crucial aspect of these models is an
expectation that, because they are treated as Gaussian process models optimized
using the marginal likelihood, they are protected from overfitting. However, we
identify situations where this is not the case. We explore this behavior,
explain its origins and consider how it applies to real datasets. Through
careful experimentation on the UCI, CIFAR-10, and the UTKFace datasets, we find
that the overfitting from overparameterized maximum marginal likelihood, in
which the model is "somewhat Bayesian", can in certain scenarios be worse than
that from not being Bayesian at all. We explain how and when DKL can still be
successful by investigating optimization dynamics. We also find that failures
of DKL can be rectified by a fully Bayesian treatment, which leads to the
desired performance improvements over standard neural networks and Gaussian
processes.
