  Video understanding calls for a model to learn the characteristic interplay
between static scene content and its dynamics: Given an image, the model must
be able to predict a future progression of the portrayed scene and, conversely,
a video should be explained in terms of its static image content and all the
remaining characteristics not present in the initial frame. This naturally
suggests a bijective mapping between the video domain and the static content as
well as residual information. In contrast to common stochastic image-to-video
synthesis, such a model does not merely generate arbitrary videos progressing
the initial image. Given this image, it rather provides a one-to-one mapping
between the residual vectors and the video with stochastic outcomes when
sampling. The approach is naturally implemented using a conditional invertible
neural network (cINN) that can explain videos by independently modelling static
and other video characteristics, thus laying the basis for controlled video
synthesis. Experiments on four diverse video datasets demonstrate the
effectiveness of our approach in terms of both the quality and diversity of the
synthesized results. Our project page is available at https://bit.ly/3t66bnU.
