  Deep Gaussian processes (DGPs) can model complex marginal densities as well
as complex mappings. Non-Gaussian marginals are essential for modelling
real-world data, and can be generated from the DGP by incorporating
uncorrelated variables to the model. Previous work on DGP models has introduced
noise additively and used variational inference with a combination of sparse
Gaussian processes and mean-field Gaussians for the approximate posterior.
Additive noise attenuates the signal, and the Gaussian form of variational
distribution may lead to an inaccurate posterior. We instead incorporate noisy
variables as latent covariates, and propose a novel importance-weighted
objective, which leverages analytic results and provides a mechanism to trade
off computation for improved accuracy. Our results demonstrate that the
importance-weighted objective works well in practice and consistently
outperforms classical variational inference, especially for deeper models.
