  Predictive learning uses a known state to generate a future state over a
period of time. It is a challenging task to predict spatiotemporal sequence
because the spatiotemporal sequence varies both in time and space. The
mainstream method is to model spatial and temporal structures at the same time
using RNN-based or transformer-based architecture, and then generates future
data by using learned experience in the way of auto-regressive. The method of
learning spatial and temporal features simultaneously brings a lot of
parameters to the model, which makes the model difficult to be convergent. In
this paper, a modular design is proposed, which decomposes spatiotemporal
sequence model into two modules: a spatial encoder-decoder and a predictor.
These two modules can extract spatial features and predict future data
respectively. The spatial encoder-decoder maps the data into a latent embedding
space and generates data from the latent space while the predictor forecasts
future embedding from past. By applying the design to the current research and
performing experiments on KTH-Action and MovingMNIST datasets, we both improve
computational performance and obtain state-of-the-art results.
