  Realistic image restoration is a crucial task in computer vision, and the use
of diffusion-based models for image restoration has garnered significant
attention due to their ability to produce realistic results. However, the
quality of the generated images is still a significant challenge due to the
severity of image degradation and the uncontrollability of the diffusion model.
In this work, we delve into the potential of utilizing pre-trained stable
diffusion for image restoration and propose MRIR, a diffusion-based restoration
method with multimodal insights. Specifically, we explore the problem from two
perspectives: textual level and visual level. For the textual level, we harness
the power of the pre-trained multimodal large language model to infer
meaningful semantic information from low-quality images. Furthermore, we employ
the CLIP image encoder with a designed Refine Layer to capture image details as
a supplement. For the visual level, we mainly focus on the pixel level control.
Thus, we utilize a Pixel-level Processor and ControlNet to control spatial
structures. Finally, we integrate the aforementioned control information into
the denoising U-Net using multi-level attention mechanisms and realize
controllable image restoration with multimodal insights. The qualitative and
quantitative results demonstrate our method's superiority over other
state-of-the-art methods on both synthetic and real-world datasets.
