  Contextual information plays an important role in action recognition. Local
operations have difficulty to model the relation between two elements with a
long-distance interval. However, directly modeling the contextual information
between any two points brings huge cost in computation and memory, especially
for action recognition, where there is an additional temporal dimension.
Inspired from 2D criss-cross attention used in segmentation task, we propose a
recurrent 3D criss-cross attention (RCCA-3D) module to model the dense
long-range spatiotemporal contextual information in video for action
recognition. The global context is factorized into sparse relation maps. We
model the relationship between points in the same line along the direction of
horizon, vertical and depth at each time, which forms a 3D criss-cross
structure, and duplicate the same operation with recurrent mechanism to
transmit the relation between points in a line to a plane finally to the whole
spatiotemporal space. Compared with the non-local method, the proposed RCCA-3D
module reduces the number of parameters and FLOPs by 25% and 30% for video
context modeling. We evaluate the performance of RCCA-3D with two latest action
recognition networks on three datasets and make a thorough analysis of the
architecture, obtaining the optimal way to factorize and fuse the relation
maps. Comparisons with other state-of-the-art methods demonstrate the
effectiveness and efficiency of our model.
