  We introduce a new video synthesis task: synthesizing time lapse videos
depicting how a given painting might have been created. Artists paint using
unique combinations of brushes, strokes, and colors. There are often many
possible ways to create a given painting. Our goal is to learn to capture this
rich range of possibilities.
  Creating distributions of long-term videos is a challenge for learning-based
video synthesis methods. We present a probabilistic model that, given a single
image of a completed painting, recurrently synthesizes steps of the painting
process. We implement this model as a convolutional neural network, and
introduce a novel training scheme to enable learning from a limited dataset of
painting time lapses. We demonstrate that this model can be used to sample many
time steps, enabling long-term stochastic video synthesis. We evaluate our
method on digital and watercolor paintings collected from video websites, and
show that human raters find our synthetic videos to be similar to time lapse
videos produced by real artists. Our code is available at
https://xamyzhao.github.io/timecraft.
