  Deep Gaussian Process (DGP) as a model prior in Bayesian learning intuitively
exploits the expressive power in function composition. DGPs also offer diverse
modeling capabilities, but inference is challenging because marginalization in
latent function space is not tractable. With Bochner's theorem, DGP with
squared exponential kernel can be viewed as a deep trigonometric network
consisting of the random feature layers, sine and cosine activation units, and
random weight layers. In the wide limit with a bottleneck, we show that the
weight space view yields the same effective covariance functions which were
obtained previously in function space. Also, varying the prior distributions
over network parameters is equivalent to employing different kernels. As such,
DGPs can be translated into the deep bottlenecked trig networks, with which the
exact maximum a posteriori estimation can be obtained. Interestingly, the
network representation enables the study of DGP's neural tangent kernel, which
may also reveal the mean of the intractable predictive distribution.
Statistically, unlike the shallow networks, deep networks of finite width have
covariance deviating from the limiting kernel, and the inner and outer widths
may play different roles in feature learning. Numerical simulations are present
to support our findings.
