  During image editing, existing deep generative models tend to re-synthesize
the entire output from scratch, including the unedited regions. This leads to a
significant waste of computation, especially for minor editing operations. In
this work, we present Spatially Sparse Inference (SSI), a general-purpose
technique that selectively performs computation for edited regions and
accelerates various generative models, including both conditional GANs and
diffusion models. Our key observation is that users prone to gradually edit the
input image. This motivates us to cache and reuse the feature maps of the
original image. Given an edited image, we sparsely apply the convolutional
filters to the edited regions while reusing the cached features for the
unedited areas. Based on our algorithm, we further propose Sparse Incremental
Generative Engine (SIGE) to convert the computation reduction to latency
reduction on off-the-shelf hardware. With about $1\%$-area edits, SIGE
accelerates DDPM by $3.0\times$ on NVIDIA RTX 3090 and $4.6\times$ on Apple M1
Pro GPU, Stable Diffusion by $7.2\times$ on 3090, and GauGAN by $5.6\times$ on
3090 and $5.2\times$ on M1 Pro GPU. Compared to our conference version, we
extend SIGE to accommodate attention layers and apply it to Stable Diffusion.
Additionally, we offer support for Apple M1 Pro GPU and include more results
with large and sequential edits.
