  In this paper, we introduce a privacy-preserving stable diffusion framework
leveraging homomorphic encryption, called HE-Diffusion, which primarily focuses
on protecting the denoising phase of the diffusion process. HE-Diffusion is a
tailored encryption framework specifically designed to align with the unique
architecture of stable diffusion, ensuring both privacy and functionality. To
address the inherent computational challenges, we propose a novel
min-distortion method that enables efficient partial image encryption,
significantly reducing the overhead without compromising the model's output
quality. Furthermore, we adopt a sparse tensor representation to expedite
computational operations, enhancing the overall efficiency of the
privacy-preserving diffusion process. We successfully implement HE-based
privacy-preserving stable diffusion inference. The experimental results show
that HE-Diffusion achieves 500 times speedup compared with the baseline method,
and reduces time cost of the homomorphically encrypted inference to the minute
level. Both the performance and accuracy of the HE-Diffusion are on par with
the plaintext counterpart. Our approach marks a significant step towards
integrating advanced cryptographic techniques with state-of-the-art generative
models, paving the way for privacy-preserving and efficient image generation in
critical applications.
