  This work aims to improve the applicability of diffusion models in realistic
image restoration. Specifically, we enhance the diffusion model in several
aspects such as network architecture, noise level, denoising steps, training
image size, and optimizer/scheduler. We show that tuning these hyperparameters
allows us to achieve better performance on both distortion and perceptual
scores. We also propose a U-Net based latent diffusion model which performs
diffusion in a low-resolution latent space while preserving high-resolution
information from the original input for the decoding process. Compared to the
previous latent-diffusion model which trains a VAE-GAN to compress the image,
our proposed U-Net compression strategy is significantly more stable and can
recover highly accurate images without relying on adversarial optimization.
Importantly, these modifications allow us to apply diffusion models to various
image restoration tasks, including real-world shadow removal, HR
non-homogeneous dehazing, stereo super-resolution, and bokeh effect
transformation. By simply replacing the datasets and slightly changing the
noise network, our model, named Refusion, is able to deal with large-size
images (e.g., 6000 x 4000 x 3 in HR dehazing) and produces good results on all
the above restoration problems. Our Refusion achieves the best perceptual
performance in the NTIRE 2023 Image Shadow Removal Challenge and wins 2nd place
overall.
