  In this work, we propose a modeling technique for jointly training image and
video generation models by simultaneously learning to map latent variables with
a fixed prior onto real images and interpolate over images to generate videos.
The proposed approach models the variations in representations using residual
vectors encoding the change at each time step over a summary vector for the
entire video. We utilize the technique to jointly train an image generation
model with a fixed prior along with a video generation model lacking
constraints such as disentanglement. The joint training enables the image
generator to exploit temporal information while the video generation model
learns to flexibly share information across frames. Moreover, experimental
results verify our approach's compatibility with pre-training on videos or
images and training on datasets containing a mixture of both. A comprehensive
set of quantitative and qualitative evaluations reveal the improvements in
sample quality and diversity over both video generation and image generation
baselines. We further demonstrate the technique's capabilities of exploiting
similarity in features across frames by applying it to a model based on
decomposing the video into motion and content. The proposed model allows minor
variations in content across frames while maintaining the temporal dependence
through latent vectors encoding the pose or motion features.
