  We define deep kernel processes in which positive definite Gram matrices are
progressively transformed by nonlinear kernel functions and by sampling from
(inverse) Wishart distributions. Remarkably, we find that deep Gaussian
processes (DGPs), Bayesian neural networks (BNNs), infinite BNNs, and infinite
BNNs with bottlenecks can all be written as deep kernel processes. For DGPs the
equivalence arises because the Gram matrix formed by the inner product of
features is Wishart distributed, and as we show, standard isotropic kernels can
be written entirely in terms of this Gram matrix -- we do not need knowledge of
the underlying features. We define a tractable deep kernel process, the deep
inverse Wishart process, and give a doubly-stochastic inducing-point
variational inference scheme that operates on the Gram matrices, not on the
features, as in DGPs. We show that the deep inverse Wishart process gives
superior performance to DGPs and infinite BNNs on standard fully-connected
baselines.
