  Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is
challenging as it typically struggles to make an appropriate trade-off between
its training data distribution and the target distribution, i.e., learning a
novel concept with only a few target images to achieve personalization
(aligning with the personalized target) while preserving text editability
(aligning with diverse text prompts). In this paper, we propose PaRa, an
effective and efficient Parameter Rank Reduction approach for T2I model
personalization by explicitly controlling the rank of the diffusion model
parameters to restrict its initial diverse generation space into a small and
well-balanced target space. Our design is motivated by the fact that taming a
T2I model toward a novel concept such as a specific art style implies a small
generation space. To this end, by reducing the rank of model parameters during
finetuning, we can effectively constrain the space of the denoising sampling
trajectories towards the target. With comprehensive experiments, we show that
PaRa achieves great advantages over existing finetuning approaches on
single/multi-subject generation as well as single-image editing. Notably,
compared to the prevailing fine-tuning technique LoRA, PaRa achieves better
parameter efficiency (2x fewer learnable parameters) and much better target
image alignment.
