  In image restoration (IR), leveraging semantic priors from segmentation
models has been a common approach to improve performance. The recent segment
anything model (SAM) has emerged as a powerful tool for extracting advanced
semantic priors to enhance IR tasks. However, the computational cost of SAM is
prohibitive for IR, compared to existing smaller IR models. The incorporation
of SAM for extracting semantic priors considerably hampers the model inference
efficiency. To address this issue, we propose a general framework to distill
SAM's semantic knowledge to boost exiting IR models without interfering with
their inference process. Specifically, our proposed framework consists of the
semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD)
scheme. SPF fuses two kinds of information between the restored image predicted
by the original IR model and the semantic mask predicted by SAM for the refined
restored image. SPD leverages a self-distillation manner to distill the fused
semantic priors to boost the performance of original IR models. Additionally,
we design a semantic-guided relation (SGR) module for SPD, which ensures
semantic feature representation space consistency to fully distill the priors.
We demonstrate the effectiveness of our framework across multiple IR models and
tasks, including deraining, deblurring, and denoising.
