  Feature shifts have been shown to be useful for action recognition with
CNN-based models since Temporal Shift Module (TSM) was proposed. It is based on
frame-wise feature extraction with late fusion, and layer features are shifted
along the time direction for the temporal interaction. TokenShift, a recent
model based on Vision Transformer (ViT), also uses the temporal feature shift
mechanism, which, however, does not fully exploit the structure of Multi-head
Self-Attention (MSA) in ViT. In this paper, we propose Multi-head
Self/Cross-Attention (MSCA), which fully utilizes the attention structure.
TokenShift is based on a frame-wise ViT with features temporally shifted with
successive frames (at time t+1 and t-1). In contrast, the proposed MSCA
replaces MSA in the frame-wise ViT, and some MSA heads attend to successive
frames instead of the current frame. The computation cost is the same as the
frame-wise ViT and TokenShift as it simply changes the target to which the
attention is taken. There is a choice about which of key, query, and value are
taken from the successive frames, then we experimentally compared these
variants with Kinetics400. We also investigate other variants in which the
proposed MSCA is used along the patch dimension of ViT, instead of the head
dimension. Experimental results show that a variant, MSCA-KV, shows the best
performance and is better than TokenShift by 0.1% and then ViT by 1.2%.
