  Standard infinite-width limits of neural networks sacrifice the ability for
intermediate layers to learn representations from data. Recent work (A theory
of representation learning gives a deep generalisation of kernel methods, Yang
et al. 2023) modified the Neural Network Gaussian Process (NNGP) limit of
Bayesian neural networks so that representation learning is retained.
Furthermore, they found that applying this modified limit to a deep Gaussian
process gives a practical learning algorithm which they dubbed the deep kernel
machine (DKM). However, they only considered the simplest possible setting:
regression in small, fully connected networks with e.g. 10 input features.
Here, we introduce convolutional deep kernel machines. This required us to
develop a novel inter-domain inducing point approximation, as well as
introducing and experimentally assessing a number of techniques not previously
seen in DKMs, including analogues to batch normalisation, different
likelihoods, and different types of top-layer. The resulting model trains in
roughly 77 GPU hours, achieving around 99% test accuracy on MNIST, 72% on
CIFAR-100, and 92.7% on CIFAR-10, which is SOTA for kernel methods.
