  Text-to-image (T2I) customization aims to create images that embody specific
visual concepts delineated in textual descriptions. However, existing works
still face a main challenge, concept overfitting. To tackle this challenge, we
first analyze overfitting, categorizing it into concept-agnostic overfitting,
which undermines non-customized concept knowledge, and concept-specific
overfitting, which is confined to customize on limited modalities, i.e,
backgrounds, layouts, styles. To evaluate the overfitting degree, we further
introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to
measure the distribution changes of non-customized and customized concept
respectively. Drawing from the analysis, we propose Infusion, a T2I
customization method that enables the learning of target concepts to avoid
being constrained by limited training modalities, while preserving
non-customized knowledge. Remarkably, Infusion achieves this feat with
remarkable efficiency, requiring a mere 11KB of trained parameters. Extensive
experiments also demonstrate that our approach outperforms state-of-the-art
methods in both single and multi-concept customized generation.
