  Diffusion models equipped with language models demonstrate excellent
controllability in image generation tasks, allowing image processing to adhere
to human instructions. However, the lack of diverse instruction-following data
hampers the development of models that effectively recognize and execute
user-customized instructions, particularly in low-level tasks. Moreover, the
stochastic nature of the diffusion process leads to deficiencies in image
generation or editing tasks that require the detailed preservation of the
generated images. To address these limitations, we propose PromptFix, a
comprehensive framework that enables diffusion models to follow human
instructions to perform a wide variety of image-processing tasks. First, we
construct a large-scale instruction-following dataset that covers comprehensive
image-processing tasks, including low-level tasks, image editing, and object
creation. Next, we propose a high-frequency guidance sampling method to
explicitly control the denoising process and preserve high-frequency details in
unprocessed areas. Finally, we design an auxiliary prompting adapter, utilizing
Vision-Language Models (VLMs) to enhance text prompts and improve the model's
task generalization. Experimental results show that PromptFix outperforms
previous methods in various image-processing tasks. Our proposed model also
achieves comparable inference efficiency with these baseline models and
exhibits superior zero-shot capabilities in blind restoration and combination
tasks. The dataset and code will be aviliable at
https://github.com/yeates/PromptFix.
