  Our work explores the task of generating future sensor observations
conditioned on the past. We are motivated by `predictive coding' concepts from
neuroscience as well as robotic applications such as self-driving vehicles.
Predictive video modeling is challenging because the future may be multi-modal
and learning at scale remains computationally expensive for video processing.
To address both challenges, our key insight is to leverage the large-scale
pretraining of image diffusion models which can handle multi-modality. We
repurpose image models for video prediction by conditioning on new frame
timestamps. Such models can be trained with videos of both static and dynamic
scenes. To allow them to be trained with modestly-sized datasets, we introduce
invariances by factoring out illumination and texture by forcing the model to
predict (pseudo) depth, readily obtained for in-the-wild videos via
off-the-shelf monocular depth networks. In fact, we show that simply modifying
networks to predict grayscale pixels already improves the accuracy of video
prediction. Given the extra controllability with timestamp conditioning, we
propose sampling schedules that work better than the traditional autoregressive
and hierarchical sampling strategies. Motivated by probabilistic metrics from
the object forecasting literature, we create a benchmark for video prediction
on a diverse set of videos spanning indoor and outdoor scenes and a large
vocabulary of objects. Our experiments illustrate the effectiveness of learning
to condition on timestamps, and show the importance of predicting the future
with invariant modalities.
