  We introduce a general framework for visual forecasting, which directly
imitates visual sequences without additional supervision. As a result, our
model can be applied at several semantic levels and does not require any domain
knowledge or handcrafted features. We achieve this by formulating visual
forecasting as an inverse reinforcement learning (IRL) problem, and directly
imitate the dynamics in natural sequences from their raw pixel values. The key
challenge is the high-dimensional and continuous state-action space that
prohibits the application of previous IRL algorithms. We address this
computational bottleneck by extending recent progress in model-free imitation
with trainable deep feature representations, which (1) bypasses the exhaustive
state-action pair visits in dynamic programming by using a dual formulation and
(2) avoids explicit state sampling at gradient computation using a deep feature
reparametrization. This allows us to apply IRL at scale and directly imitate
the dynamics in high-dimensional continuous visual sequences from the raw pixel
values. We evaluate our approach at three different level-of-abstraction, from
low level pixels to higher level semantics: future frame generation, action
anticipation, visual story forecasting. At all levels, our approach outperforms
existing methods.
