  Spatiotemporal predictive learning, which predicts future frames through
historical prior knowledge with the aid of deep learning, is widely used in
many fields. Previous work essentially improves the model performance by
widening or deepening the network, but it also brings surging memory overhead,
which seriously hinders the development and application of this technology. In
order to improve the performance without increasing memory consumption, we
focus on scale, which is another dimension to improve model performance but
with low memory requirement. The effectiveness has been widely demonstrated in
many CNN-based tasks such as image classification and semantic segmentation,
but it has not been fully explored in recent RNN models. In this paper,
learning from the benefit of multi-scale, we propose a general framework named
Multi-Scale RNN (MS-RNN) to boost recent RNN models for spatiotemporal
predictive learning. We verify the MS-RNN framework by thorough theoretical
analyses and exhaustive experiments, where the theory focuses on memory
reduction and performance improvement while the experiments employ eight RNN
models (ConvLSTM, TrajGRU, PredRNN, PredRNN++, MIM, MotionRNN, PredRNN-V2, and
PrecipLSTM) and four datasets (Moving MNIST, TaxiBJ, KTH, and Germany). The
results show the efficiency that RNN models incorporating our framework have
much lower memory cost but better performance than before. Our code is released
at \url{https://github.com/mazhf/MS-RNN}.
