  We study on image super-resolution (SR), which aims to recover realistic
textures from a low-resolution (LR) image. Recent progress has been made by
taking high-resolution images as references (Ref), so that relevant textures
can be transferred to LR images. However, existing SR approaches neglect to use
attention mechanisms to transfer high-resolution (HR) textures from Ref images,
which limits these approaches in challenging cases. In this paper, we propose a
novel Texture Transformer Network for Image Super-Resolution (TTSR), in which
the LR and Ref images are formulated as queries and keys in a transformer,
respectively. TTSR consists of four closely-related modules optimized for image
generation tasks, including a learnable texture extractor by DNN, a relevance
embedding module, a hard-attention module for texture transfer, and a
soft-attention module for texture synthesis. Such a design encourages joint
feature learning across LR and Ref images, in which deep feature
correspondences can be discovered by attention, and thus accurate texture
features can be transferred. The proposed texture transformer can be further
stacked in a cross-scale way, which enables texture recovery from different
levels (e.g., from 1x to 4x magnification). Extensive experiments show that
TTSR achieves significant improvements over state-of-the-art approaches on both
quantitative and qualitative evaluations.
