  Text-to-image diffusion models (SD) exhibit significant advancements while
requiring extensive computational resources. Though many acceleration methods
have been proposed, they suffer from generation quality degradation or extra
training cost generalizing to new fine-tuned models. To address these
limitations, we propose a novel and universal Stable-Diffusion (SD)
acceleration module called SpeedUpNet(SUN). SUN can be directly plugged into
various fine-tuned SD models without extra training. This technique utilizes
cross-attention layers to learn the relative offsets in the generated image
results between negative and positive prompts achieving classifier-free
guidance distillation with negative prompts controllable, and introduces a
Multi-Step Consistency (MSC) loss to ensure a harmonious balance between
reducing inference steps and maintaining consistency in the generated output.
Consequently, SUN significantly reduces the number of inference steps to just 4
steps and eliminates the need for classifier-free guidance. It leads to an
overall speedup of more than 10 times for SD models compared to the
state-of-the-art 25-step DPM-solver++, and offers two extra advantages: (1)
classifier-free guidance distillation with controllable negative prompts and
(2) seamless integration into various fine-tuned Stable-Diffusion models
without training. The effectiveness of the SUN has been verified through
extensive experimentation. Project Page:
https://williechai.github.io/speedup-plugin-for-stable-diffusions.github.io
