  Learning to capture long-range relations is fundamental to image/video
recognition. Existing CNN models generally rely on increasing depth to model
such relations which is highly inefficient. In this work, we propose the
"double attention block", a novel component that aggregates and propagates
informative global features from the entire spatio-temporal space of input
images/videos, enabling subsequent convolution layers to access features from
the entire space efficiently. The component is designed with a double attention
mechanism in two steps, where the first step gathers features from the entire
space into a compact set through second-order attention pooling and the second
step adaptively selects and distributes features to each location via another
attention. The proposed double attention block is easy to adopt and can be
plugged into existing deep neural networks conveniently. We conduct extensive
ablation studies and experiments on both image and video recognition tasks for
evaluating its performance. On the image recognition task, a ResNet-50 equipped
with our double attention blocks outperforms a much larger ResNet-152
architecture on ImageNet-1k dataset with over 40% less the number of parameters
and less FLOPs. On the action recognition task, our proposed model achieves the
state-of-the-art results on the Kinetics and UCF-101 datasets with
significantly higher efficiency than recent works.
