  In this work, we use Deep Gaussian Processes (DGPs) as statistical surrogates
for stochastic processes with complex distributions. Conventional inferential
methods for DGP models can suffer from high computational complexity as they
require large-scale operations with kernel matrices for training and inference.
In this work, we propose an efficient scheme for accurate inference and
efficient training based on a range of Gaussian Processes, called the Tensor
Markov Gaussian Processes (TMGP). We construct an induced approximation of TMGP
referred to as the hierarchical expansion. Next, we develop a deep TMGP (DTMGP)
model as the composition of multiple hierarchical expansion of TMGPs. The
proposed DTMGP model has the following properties: (1) the outputs of each
activation function are deterministic while the weights are chosen
independently from standard Gaussian distribution; (2) in training or
prediction, only polylog(M) (out of M) activation functions have non-zero
outputs, which significantly boosts the computational efficiency. Our numerical
experiments on synthetic models and real datasets show the superior
computational efficiency of DTMGP over existing DGP models.
