  Neural-net-induced Gaussian process (NNGP) regression inherits both the high
expressivity of deep neural networks (deep NNs) as well as the uncertainty
quantification property of Gaussian processes (GPs). We generalize the current
NNGP to first include a larger number of hyperparameters and subsequently train
the model by maximum likelihood estimation. Unlike previous works on NNGP that
targeted classification, here we apply the generalized NNGP to function
approximation and to solving partial differential equations (PDEs).
Specifically, we develop an analytical iteration formula to compute the
covariance function of GP induced by deep NN with an error-function
nonlinearity. We compare the performance of the generalized NNGP for function
approximations and PDE solutions with those of GPs and fully-connected NNs. We
observe that for smooth functions the generalized NNGP can yield the same order
of accuracy with GP, while both NNGP and GP outperform deep NN. For non-smooth
functions, the generalized NNGP is superior to GP and comparable or superior to
deep NN.
