  Gaussian processes (GPs) are nonparametric priors over functions. Fitting a
GP implies computing a posterior distribution of functions consistent with the
observed data. Similarly, deep Gaussian processes (DGPs) should allow us to
compute a posterior distribution of compositions of multiple functions giving
rise to the observations. However, exact Bayesian inference is intractable for
DGPs, motivating the use of various approximations. We show that the
application of simplifying mean-field assumptions across the hierarchy leads to
the layers of a DGP collapsing to near-deterministic transformations. We argue
that such an inference scheme is suboptimal, not taking advantage of the
potential of the model to discover the compositional structure in the data. To
address this issue, we examine alternative variational inference schemes
allowing for dependencies across different layers and discuss their advantages
and limitations.
