  Benefiting from the event-driven and sparse spiking characteristics of the
brain, spiking neural networks (SNNs) are becoming an energy-efficient
alternative to artificial neural networks (ANNs). However, the performance gap
between SNNs and ANNs has been a great hindrance to deploying SNNs ubiquitously
for a long time. To leverage the full potential of SNNs, we study the effect of
attention mechanisms in SNNs. We first present our idea of attention with a
plug-and-play kit, termed the Multi-dimensional Attention (MA). Then, a new
attention SNN architecture with end-to-end training called "MA-SNN" is
proposed, which infers attention weights along the temporal, channel, as well
as spatial dimensions separately or simultaneously. Based on the existing
neuroscience theories, we exploit the attention weights to optimize membrane
potentials, which in turn regulate the spiking response in a data-dependent
way. At the cost of negligible additional parameters, MA facilitates vanilla
SNNs to achieve sparser spiking activity, better performance, and energy
efficiency concurrently. Experiments are conducted in event-based DVS128
Gesture/Gait action recognition and ImageNet-1k image classification. On
Gesture/Gait, the spike counts are reduced by 84.9%/81.6%, and the task
accuracy and energy efficiency are improved by 5.9%/4.7% and
3.4$\times$/3.2$\times$. On ImageNet-1K, we achieve top-1 accuracy of 75.92%
and 77.08% on single/4-step Res-SNN-104, which are state-of-the-art results in
SNNs. To our best knowledge, this is for the first time, that the SNN community
achieves comparable or even better performance compared with its ANN
counterpart in the large-scale dataset. Our work lights up SNN's potential as a
general backbone to support various applications for SNNs, with a great balance
between effectiveness and efficiency.
