  Convolutional neural networks have allowed remarkable advances in single
image super-resolution (SISR) over the last decade. Among recent advances in
SISR, attention mechanisms are crucial for high-performance SR models. However,
the attention mechanism remains unclear on why and how it works in SISR. In
this work, we attempt to quantify and visualize attention mechanisms in SISR
and show that not all attention modules are equally beneficial. We then propose
attention in attention network (A$^2$N) for more efficient and accurate SISR.
Specifically, A$^2$N consists of a non-attention branch and a coupling
attention branch. A dynamic attention module is proposed to generate weights
for these two branches to suppress unwanted attention adjustments dynamically,
where the weights change adaptively according to the input features. This
allows attention modules to specialize to beneficial examples without otherwise
penalties and thus greatly improve the capacity of the attention network with
few parameters overhead. Experimental results demonstrate that our final model
A$^2$N could achieve superior trade-off performances comparing with
state-of-the-art networks of similar sizes. Codes are available at
https://github.com/haoyuc/A2N.
