  Conventional variational autoencoders fail in modeling correlations between
data points due to their use of factorized priors. Amortized Gaussian process
inference through GP-VAEs has led to significant improvements in this regard,
but is still inhibited by the intrinsic complexity of exact GP inference. We
improve the scalability of these methods through principled sparse inference
approaches. We propose a new scalable GP-VAE model that outperforms existing
approaches in terms of runtime and memory footprint, is easy to implement, and
allows for joint end-to-end optimization of all components.
