  Variational autoencoders (VAEs) are a powerful class of deep generative
latent variable model for unsupervised representation learning on
high-dimensional data. To ensure computational tractability, VAEs are often
implemented with a univariate standard Gaussian prior and a mean-field Gaussian
variational posterior distribution. This results in a vector-valued latent
variables that are agnostic to the original data structure which might be
highly correlated across and within multiple dimensions. We propose a
tensor-variate extension to the VAE framework, the tensor-variate Gaussian
process prior variational autoencoder (tvGP-VAE), which replaces the standard
univariate Gaussian prior and posterior distributions with tensor-variate
Gaussian processes. The tvGP-VAE is able to explicitly model correlation
structures via the use of kernel functions over the dimensions of tensor-valued
latent variables. Using spatiotemporally correlated image time series as an
example, we show that the choice of which correlation structures to explicitly
represent in the latent space has a significant impact on model performance in
terms of reconstruction.
