  Spatiotemporal predictive learning aims to generate future frames by learning
from historical frames. In this paper, we investigate existing methods and
present a general framework of spatiotemporal predictive learning, in which the
spatial encoder and decoder capture intra-frame features and the middle
temporal module catches inter-frame correlations. While the mainstream methods
employ recurrent units to capture long-term temporal dependencies, they suffer
from low computational efficiency due to their unparallelizable architectures.
To parallelize the temporal module, we propose the Temporal Attention Unit
(TAU), which decomposes the temporal attention into intra-frame statical
attention and inter-frame dynamical attention. Moreover, while the mean squared
error loss focuses on intra-frame errors, we introduce a novel differential
divergence regularization to take inter-frame variations into account.
Extensive experiments demonstrate that the proposed method enables the derived
model to achieve competitive performance on various spatiotemporal prediction
benchmarks.
