  Advanced diffusion-based Text-to-Image (T2I) models, such as the Stable
Diffusion Model, have made significant progress in generating diverse and
high-quality images using text prompts alone. However, when non-famous users
require personalized image generation for their identities (IDs), the T2I
models fail to accurately generate their ID-related images. The main problem is
that pre-trained T2I models do not learn the mapping between the new ID prompts
and their corresponding visual content. The previous methods either failed to
accurately fit the face region or lost the interactive generative ability with
other existing concepts in T2I models. In other words, they are unable to
generate T2I-aligned and semantic-fidelity images for the given prompts with
other concepts such as scenes (``Eiffel Tower''), actions (``holding a
basketball''), and facial attributes (``eyes closed''). In this paper, we focus
on inserting accurate and interactive ID embedding into the Stable Diffusion
Model for semantic-fidelity personalized generation. We address this challenge
from two perspectives: face-wise region fitting and semantic-fidelity token
optimization. Specifically, we first visualize the attention overfit problem
and propose a face-wise attention loss to fit the face region instead of
entangling ID-unrelated information, such as face layout and background. This
key trick significantly enhances the ID accuracy and interactive generative
ability with other existing concepts. Then, we optimize one ID representation
as multiple per-stage tokens where each token contains two disentangled
features. This expansion of the textual conditioning space improves
semantic-fidelity control. Extensive experiments validate that our results
exhibit superior ID accuracy, text-based manipulation ability, and
generalization compared to previous methods.
