  Image super-resolution (SR) serves as a fundamental tool for the processing
and transmission of multimedia data. Recently, Transformer-based models have
achieved competitive performances in image SR. They divide images into
fixed-size patches and apply self-attention on these patches to model
long-range dependencies among pixels. However, this architecture design is
originated for high-level vision tasks, which lacks design guideline from SR
knowledge. In this paper, we aim to design a new attention block whose insights
are from the interpretation of Local Attribution Map (LAM) for SR networks.
Specifically, LAM presents a hierarchical importance map where the most
important pixels are located in a fine area of a patch and some less important
pixels are spread in a coarse area of the whole image. To access pixels in the
coarse area, instead of using a very large patch size, we propose a lightweight
Global Pixel Access (GPA) module that applies cross-attention with the most
similar patch in an image. In the fine area, we use an Intra-Patch
Self-Attention (IPSA) module to model long-range pixel dependencies in a local
patch, and then a $3\times3$ convolution is applied to process the finest
details. In addition, a Cascaded Patch Division (CPD) strategy is proposed to
enhance perceptual quality of recovered images. Extensive experiments suggest
that our method outperforms state-of-the-art lightweight SR methods by a large
margin. Code is available at https://github.com/passerer/HPINet.
