  Prediction of future states of the environment and interacting agents is a
key competence required for autonomous agents to operate successfully in the
real world. Prior work for structured sequence prediction based on latent
variable models imposes a uni-modal standard Gaussian prior on the latent
variables. This induces a strong model bias which makes it challenging to fully
capture the multi-modality of the distribution of the future states. In this
work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our
novel conditional normalizing flow based prior to capture complex multi-modal
conditional distributions for effective structured sequence prediction.
Moreover, we propose two novel regularization schemes which stabilizes training
and deals with posterior collapse for stable training and better fit to the
target data distribution. Our experiments on three multi-modal structured
sequence prediction datasets -- MNIST Sequences, Stanford Drone and HighD --
show that the proposed method obtains state of art results across different
evaluation metrics.
