  The Stable Diffusion Model (SDM) is a prevalent and effective model for
text-to-image (T2I) and image-to-image (I2I) generation. Despite various
attempts at sampler optimization, model distillation, and network
quantification, these approaches typically maintain the original network
architecture. The extensive parameter scale and substantial computational
demands have limited research into adjusting the model architecture. This study
focuses on reducing redundant computation in SDM and optimizes the model
through both tuning and tuning-free methods. 1) For the tuning method, we
design a model assembly strategy to reconstruct a lightweight model while
preserving performance through distillation. Second, to mitigate performance
loss due to pruning, we incorporate multi-expert conditional convolution
(ME-CondConv) into compressed UNets to enhance network performance by
increasing capacity without sacrificing speed. Third, we validate the
effectiveness of the multi-UNet switching method for improving network speed.
2) For the tuning-free method, we propose a feature inheritance strategy to
accelerate inference by skipping local computations at the block, layer, or
unit level within the network structure. We also examine multiple sampling
modes for feature inheritance at the time-step level. Experiments demonstrate
that both the proposed tuning and the tuning-free methods can improve the speed
and performance of the SDM. The lightweight model reconstructed by the model
assembly strategy increases generation speed by $22.4%$, while the feature
inheritance strategy enhances the SDM generation speed by $40.0%$.
