  Probabilistic modelling has been an essential tool in medical image analysis,
especially for analyzing brain Magnetic Resonance Images (MRI). Recent deep
learning techniques for estimating high-dimensional distributions, in
particular Variational Autoencoders (VAEs), opened up new avenues for
probabilistic modeling. Modelling of volumetric data has remained a challenge,
however, because constraints on available computation and training data make it
difficult effectively leverage VAEs, which are well-developed for 2D images. We
propose a method to model 3D MR brain volumes distribution by combining a 2D
slice VAE with a Gaussian model that captures the relationships between slices.
We do so by estimating the sample mean and covariance in the latent space of
the 2D model over the slice direction. This combined model lets us sample new
coherent stacks of latent variables to decode into slices of a volume. We also
introduce a novel evaluation method for generated volumes that quantifies how
well their segmentations match those of true brain anatomy. We demonstrate that
our proposed model is competitive in generating high quality volumes at high
resolutions according to both traditional metrics and our proposed evaluation.
