  Spatiotemporal predictive learning (ST-PL) is a hotspot with numerous
applications, such as object movement and meteorological prediction. It aims at
predicting the subsequent frames via observed sequences. However, inherent
uncertainty among consecutive frames exacerbates the difficulty in long-term
prediction. To tackle the increasing ambiguity during forecasting, we design
CMS-LSTM to focus on context correlations and multi-scale spatiotemporal flow
with details on fine-grained locals, containing two elaborate designed blocks:
Context Embedding (CE) and Spatiotemporal Expression (SE) blocks. CE is
designed for abundant context interactions, while SE focuses on multi-scale
spatiotemporal expression in hidden states. The newly introduced blocks also
facilitate other spatiotemporal models (e.g., PredRNN, SA-ConvLSTM) to produce
representative implicit features for ST-PL and improve prediction quality.
Qualitative and quantitative experiments demonstrate the effectiveness and
flexibility of our proposed method. With fewer params, CMS-LSTM outperforms
state-of-the-art methods in numbers of metrics on two representative benchmarks
and scenarios. Code is available at https://github.com/czh-98/CMS-LSTM.
