  Variational inference is a powerful tool for approximate inference, and it
has been recently applied for representation learning with deep generative
models. We develop the variational Gaussian process (VGP), a Bayesian
nonparametric variational family, which adapts its shape to match complex
posterior distributions. The VGP generates approximate posterior samples by
generating latent inputs and warping them through random non-linear mappings;
the distribution over random mappings is learned during inference, enabling the
transformed outputs to adapt to varying complexity. We prove a universal
approximation theorem for the VGP, demonstrating its representative power for
learning any model. For inference we present a variational objective inspired
by auto-encoders and perform black box inference over a wide class of models.
The VGP achieves new state-of-the-art results for unsupervised learning,
inferring models such as the deep latent Gaussian model and the recently
proposed DRAW.
