  Text-to-image (T2I) generation with Stable Diffusion models (SDMs) involves
high computing demands due to billion-scale parameters. To enhance efficiency,
recent studies have reduced sampling steps and applied network quantization
while retaining the original architectures. The lack of architectural reduction
attempts may stem from worries over expensive retraining for such massive
models. In this work, we uncover the surprising potential of block pruning and
feature distillation for low-cost general-purpose T2I. By removing several
residual and attention blocks from the U-Net of SDMs, we achieve 30%~50%
reduction in model size, MACs, and latency. We show that distillation
retraining is effective even under limited resources: using only 13 A100 days
and a tiny dataset, our compact models can imitate the original SDMs (v1.4 and
v2.1-base with over 6,000 A100 days). Benefiting from the transferred
knowledge, our BK-SDMs deliver competitive results on zero-shot MS-COCO against
larger multi-billion parameter models. We further demonstrate the applicability
of our lightweight backbones in personalized generation and image-to-image
translation. Deployment of our models on edge devices attains 4-second
inference. We hope this work can help build small yet powerful diffusion models
with feasible training budgets. Code and models can be found at:
https://github.com/Nota-NetsPresso/BK-SDM
