  We propose Self-Supervised Implicit Attention (SSIA), a new approach that
adaptively guides deep neural network models to gain attention by exploiting
the properties of the models themselves. SSIA is a novel attention mechanism
that does not require any extra parameters, computation, or memory access costs
during inference, which is in contrast to existing attention mechanism. In
short, by considering attention weights as higher-level semantic information,
we reconsidered the implementation of existing attention mechanisms and further
propose generating supervisory signals from higher network layers to guide
lower network layers for parameter updates. We achieved this by building a
self-supervised learning task using the hierarchical features of the network
itself, which only works at the training stage. To verify the effectiveness of
SSIA, we performed a particular implementation (called an SSIA block) in
convolutional neural network models and validated it on several image
classification datasets. The experimental results show that an SSIA block can
significantly improve the model performance, even outperforms many popular
attention methods that require additional parameters and computation costs,
such as Squeeze-and-Excitation and Convolutional Block Attention Module. Our
implementation will be available on GitHub.
