FaceCLIP Model Card

This repository provides the official models for the following paper:

Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis
Zichuan Liu     Liming Jiang     Qing Yan     Yumin Jia     Hao Kang     Xin Lu     Min Jin Chong

Abstract: Recent progress in text-to-image (T2I) diffusion models has greatly improved image quality and flexibility. However, a major challenge in personalized generation remains: preserving the subject’s identity (ID) while allowing diverse visual changes. We address this with a new framework for ID-preserving image generation. Instead of relying on adapter modules to inject identity features into pre-trained models, we propose a unified multi-modal encoding strategy that jointly captures identity and text information. Our method, called FaceCLIP, learns a shared embedding space for facial identity and textual semantics. Given a reference face image and a text prompt, FaceCLIP produces a joint representation that guides the generative model to synthesize images consistent with both the subject’s identity and the prompt. To train FaceCLIP, we introduce a multi-modal alignment loss that aligns features across face, text, and image domains. We then integrate FaceCLIP with existing UNet and Diffusion Transformer (DiT) architectures, forming a complete synthesis pipeline FaceCLIP-x. Compared to existing ID-preserving approaches, our method produces more photorealistic portraits with better identity retention and text alignment. Extensive experiments demonstrate that FaceCLIP-x outperforms prior methods in both qualitative and quantitative evaluations.

🔧 Installation and Usage

Please clone our GitHub code repository and follow the detailed instruction to install and use the released models for local inference.

Model Zoo

Version	Description
FaceCLIP-SDXL	SDXL base model trained with FaceCLIP-L-14 and FaceCLIP-bigG-14 encoders.
FaceT5-FLUX	FLUX.1-dev base model trained with FaceT5 encoder.

📜 Disclaimer and Licenses

The images used in this repository and related demos are sourced from consented subjects or generated by the models. These pictures are intended solely to showcase the capabilities of our research. If you have any concerns, please feel free to contact us, and we will promptly remove any inappropriate content.

Our model is released under the Creative Commons Attribution-NonCommercial 4.0 International Public License for academic research purposes only. Any manual or automatic downloading of the face models from the OpenAI-CLIP-L-14, OpenCLIP-bigG-14, the SDXL-base-1.0 base model, and the FLUX.1-dev base model, etc., must follow their original licenses and be used only for academic research purposes.

This research aims to positively impact the field of Generative AI. Any usage of this method must be responsible and comply with local laws. The developers do not assume any responsibility for any potential misuse.

📖 Citation

If you find FaceCLIP useful for your research or applications, please cite our paper:

@article{liu2025learning,
  title={Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis},
  author={Liu, Zichuan and Jiang, Liming and Yan, Qing and Jia, Yumin and Kang, Hao and Lu, Xin},
  journal={arXiv preprint arXiv:2504.14202},
  year={2025}
}

We also appreciate it if you could give a star ⭐ to our Github repository. Thanks a lot!