  Gaussian processes (GPs) are an attractive class of machine learning models
because of their simplicity and flexibility as building blocks of more complex
Bayesian models. Meanwhile, graph neural networks (GNNs) emerged recently as a
promising class of models for graph-structured data in semi-supervised learning
and beyond. Their competitive performance is often attributed to a proper
capturing of the graph inductive bias. In this work, we introduce this
inductive bias into GPs to improve their predictive performance for
graph-structured data. We show that a prominent example of GNNs, the graph
convolutional network, is equivalent to some GP when its layers are infinitely
wide; and we analyze the kernel universality and the limiting behavior in
depth. We further present a programmable procedure to compose covariance
kernels inspired by this equivalence and derive example kernels corresponding
to several interesting members of the GNN family. We also propose a
computationally efficient approximation of the covariance matrix for scalable
posterior inference with large-scale data. We demonstrate that these
graph-based kernels lead to competitive classification and regression
performance, as well as advantages in computation time, compared with the
respective GNNs.
