  Self-supervised prediction is a powerful mechanism to learn representations
that capture the underlying structure of the data. Despite recent progress, the
self-supervised video prediction task is still challenging. One of the critical
factors that make the task hard is motion segmentation, which is segmenting
individual objects and the background and estimating their motion separately.
In video prediction, the shape, appearance, and transformation of each object
should be understood only by predicting the next frame in pixel space. To
address this task, we propose a novel end-to-end learnable architecture that
predicts the next frame by modeling foreground and background separately while
simultaneously estimating and predicting the foreground motion using Frequency
Domain Transformer Networks. Experimental evaluations show that this yields
interpretable representations and that our approach can outperform some widely
used video prediction methods like Video Ladder Network and Predictive Gated
Pyramids on synthetic data.
