  The ability of predicting the future is important for intelligent systems,
e.g. autonomous vehicles and robots to plan early and make decisions
accordingly. Future scene parsing and optical flow estimation are two key tasks
that help agents better understand their environments as the former provides
dense semantic information, i.e. what objects will be present and where they
will appear, while the latter provides dense motion information, i.e. how the
objects will move. In this paper, we propose a novel model to simultaneously
predict scene parsing and optical flow in unobserved future video frames. To
our best knowledge, this is the first attempt in jointly predicting scene
parsing and motion dynamics. In particular, scene parsing enables structured
motion prediction by decomposing optical flow into different groups while
optical flow estimation brings reliable pixel-wise correspondence to scene
parsing. By exploiting this mutually beneficial relationship, our model shows
significantly better parsing and motion prediction results when compared to
well-established baselines and individual prediction models on the large-scale
Cityscapes dataset. In addition, we also demonstrate that our model can be used
to predict the steering angle of the vehicles, which further verifies the
ability of our model to learn latent representations of scene dynamics.
