  Video prediction is an important yet challenging problem; burdened with the
tasks of generating future frames and learning environment dynamics. Recently,
autoregressive latent video models have proved to be a powerful video
prediction tool, by separating the video prediction into two sub-problems:
pre-training an image generator model, followed by learning an autoregressive
prediction model in the latent space of the image generator. However,
successfully generating high-fidelity and high-resolution videos has yet to be
seen. In this work, we investigate how to train an autoregressive latent video
prediction model capable of predicting high-fidelity future frames with minimal
modification to existing models, and produce high-resolution (256x256) videos.
Specifically, we scale up prior models by employing a high-fidelity image
generator (VQ-GAN) with a causal transformer model, and introduce additional
techniques of top-k sampling and data augmentation to further improve video
prediction quality. Despite the simplicity, the proposed method achieves
competitive performance to state-of-the-art approaches on standard video
prediction benchmarks with fewer parameters, and enables high-resolution video
prediction on complex and large-scale datasets. Videos are available at
https://sites.google.com/view/harp-videos/home.
