  Efficient spatiotemporal modeling is an important yet challenging problem for
video action recognition. Existing state-of-the-art methods exploit neighboring
feature differences to obtain motion clues for short-term temporal modeling
with a simple convolution. However, only one local convolution is incapable of
handling various kinds of actions because of the limited receptive field.
Besides, action-irrelated noises brought by camera movement will also harm the
quality of extracted motion features. In this paper, we propose a Temporal
Saliency Integration (TSI) block, which mainly contains a Salient Motion
Excitation (SME) module and a Cross-perception Temporal Integration (CTI)
module. Specifically, SME aims to highlight the motion-sensitive area through
spatial-level local-global motion modeling, where the saliency alignment and
pyramidal motion modeling are conducted successively between adjacent frames to
capture motion dynamics with fewer noises caused by misaligned background. CTI
is designed to perform multi-perception temporal modeling through a group of
separate 1D convolutions respectively. Meanwhile, temporal interactions across
different perceptions are integrated with the attention mechanism. Through
these two modules, long short-term temporal relationships can be encoded
efficiently by introducing limited additional parameters. Extensive experiments
are conducted on several popular benchmarks (i.e., Something-Something V1 & V2,
Kinetics-400, UCF-101, and HMDB-51), which demonstrate the effectiveness of our
proposed method.
