  We address the video prediction task by putting forth a novel model that
combines (i) our recently proposed hierarchical residual vector quantized
variational autoencoder (HR-VQVAE), and (ii) a novel spatiotemporal PixelCNN
(ST-PixelCNN). We refer to this approach as a sequential hierarchical residual
learning vector quantized variational autoencoder (S-HR-VQVAE). By leveraging
the intrinsic capabilities of HR-VQVAE at modeling still images with a
parsimonious representation, combined with the ST-PixelCNN's ability at
handling spatiotemporal information, S-HR-VQVAE can better deal with chief
challenges in video prediction. These include learning spatiotemporal
information, handling high dimensional data, combating blurry prediction, and
implicit modeling of physical characteristics. Extensive experimental results
on the KTH Human Action and Moving-MNIST tasks demonstrate that our model
compares favorably against top video prediction techniques both in quantitative
and qualitative evaluations despite a much smaller model size. Finally, we
boost S-HR-VQVAE by proposing a novel training method to jointly estimate the
HR-VQVAE and ST-PixelCNN parameters.
