  Image Restoration (IR), a classic low-level vision task, has witnessed
significant advancements through deep models that effectively model global
information. Notably, the Vision Transformers (ViTs) emergence has further
propelled these advancements. When computing, the self-attention mechanism, a
cornerstone of ViTs, tends to encompass all global cues, even those from
semantically unrelated objects or regions. This inclusivity introduces
computational inefficiencies, particularly noticeable with high input
resolution, as it requires processing irrelevant information, thereby impeding
efficiency. Additionally, for IR, it is commonly noted that small segments of a
degraded image, particularly those closely aligned semantically, provide
particularly relevant information to aid in the restoration process, as they
contribute essential contextual cues crucial for accurate reconstruction. To
address these challenges, we propose boosting IR's performance by sharing the
key semantics via Transformer for IR (i.e., SemanIR) in this paper.
Specifically, SemanIR initially constructs a sparse yet comprehensive
key-semantic dictionary within each transformer stage by establishing essential
semantic connections for every degraded patch. Subsequently, this dictionary is
shared across all subsequent transformer blocks within the same stage. This
strategy optimizes attention calculation within each block by focusing
exclusively on semantically related components stored in the key-semantic
dictionary. As a result, attention calculation achieves linear computational
complexity within each window. Extensive experiments across 6 IR tasks confirm
the proposed SemanIR's state-of-the-art performance, quantitatively and
qualitatively showcasing advancements.
