  Sequential VAEs have been successfully considered for many high-dimensional
time series modelling problems, with many variant models relying on
discrete-time mechanisms such as recurrent neural networks (RNNs). On the other
hand, continuous-time methods have recently gained attraction, especially in
the context of irregularly-sampled time series, where they can better handle
the data than discrete-time methods. One such class are Gaussian process
variational autoencoders (GPVAEs), where the VAE prior is set as a Gaussian
process (GP). However, a major limitation of GPVAEs is that it inherits the
cubic computational cost as GPs, making it unattractive to practioners. In this
work, we leverage the equivalent discrete state space representation of
Markovian GPs to enable linear time GPVAE training via Kalman filtering and
smoothing. For our model, Markovian GPVAE (MGPVAE), we show on a variety of
high-dimensional temporal and spatiotemporal tasks that our method performs
favourably compared to existing approaches whilst being computationally highly
scalable.
