  We present a VAE architecture for encoding and generating high dimensional
sequential data, such as video or audio. Our deep generative model learns a
latent representation of the data which is split into a static and dynamic
part, allowing us to approximately disentangle latent time-dependent features
(dynamics) from features which are preserved over time (content). This
architecture gives us partial control over generating content and dynamics by
conditioning on either one of these sets of features. In our experiments on
artificially generated cartoon video clips and voice recordings, we show that
we can convert the content of a given sequence into another one by such content
swapping. For audio, this allows us to convert a male speaker into a female
speaker and vice versa, while for video we can separately manipulate shapes and
dynamics. Furthermore, we give empirical evidence for the hypothesis that
stochastic RNNs as latent state models are more efficient at compressing and
generating long sequences than deterministic ones, which may be relevant for
applications in video compression.
