  Long-range spatiotemporal dependencies capturing plays an essential role in
improving video features for action recognition. The non-local block inspired
by the non-local means is designed to address this challenge and have shown
excellent performance. However, the non-local block brings significant increase
in computation cost to the original network. It also lacks the ability to model
regional correlation in videos. To address the above limitations, we propose
Pyramid Non-Local (PNL) module, which extends the non-local block by
incorporating regional correlation at multiple scales through a pyramid
structured module. This extension upscales the effectiveness of non-local
operation by attending to the interaction between different regions. Empirical
results prove the effectiveness and efficiency of our PNL module, which
achieves state-of-the-art performance of 83.09% on the Mini-Kinetics dataset,
with decreased computation cost compared to the non-local block.
