  Video prediction, predicting future frames from the previous ones, has broad
applications such as autonomous driving and weather forecasting. Existing
state-of-the-art methods typically focus on extracting either spatial,
temporal, or spatiotemporal features from videos. Different feature focuses,
resulting from different network architectures, may make the resultant models
excel at some video prediction tasks but perform poorly on others. Towards a
more generic video prediction solution, we explicitly model these features in a
unified encoder-decoder framework and propose a novel simple alternating Mixer
(SIAM). The novelty of SIAM lies in the design of dimension alternating mixing
(DaMi) blocks, which can model spatial, temporal, and spatiotemporal features
through alternating the dimensions of the feature maps. Extensive experimental
results demonstrate the superior performance of the proposed SIAM on four
benchmark video datasets covering both synthetic and real-world scenarios.
