  Video action recognition (VAR) plays crucial roles in various domains such as
surveillance, healthcare, and industrial automation, making it highly
significant for the society. Consequently, it has long been a research spot in
the computer vision field. As artificial neural networks (ANNs) are
flourishing, convolution neural networks (CNNs), including 2D-CNNs and 3D-CNNs,
as well as variants of the vision transformer (ViT), have shown impressive
performance on VAR. However, they usually demand huge computational cost due to
the large data volume and heavy information redundancy introduced by the
temporal dimension. To address this challenge, some researchers have turned to
brain-inspired spiking neural networks (SNNs), such as recurrent SNNs and
ANN-converted SNNs, leveraging their inherent temporal dynamics and energy
efficiency. Yet, current SNNs for VAR also encounter limitations, such as
nontrivial input preprocessing, intricate network construction/training, and
the need for repetitive processing of the same video clip, hindering their
practical deployment. In this study, we innovatively propose the directly
trained SVFormer (Spiking Video transFormer) for VAR. SVFormer integrates local
feature extraction, global self-attention, and the intrinsic dynamics,
sparsity, and spike-driven nature of SNNs, to efficiently and effectively
extract spatio-temporal features. We evaluate SVFormer on two RGB datasets
(UCF101, NTU-RGBD60) and one neuromorphic dataset (DVS128-Gesture),
demonstrating comparable performance to the mainstream models in a more
efficient way. Notably, SVFormer achieves a top-1 accuracy of 84.03% with
ultra-low power consumption (21 mJ/video) on UCF101, which is state-of-the-art
among directly trained deep SNNs, showcasing significant advantages over prior
models.
