Improve model card: add paper info, links, and improve description

README.md

@@ -1,215 +1,88 @@
 ---
 license: openrail++
+library_name: diffusers
 tags:
 - text-to-image
 - stable-diffusion
 ---
-# SD-XL 1.0-base Model Card
-![row01](01.png)
 
-## Model
+# Conceptrol: Concept Control of Zero-shot Personalized Image Generation
 
-![pipeline](pipeline.png)
+## Model Card
 
-[SDXL](https://arxiv.org/abs/2307.01952) consists of an [ensemble of experts](https://arxiv.org/abs/2211.01324) pipeline for latent diffusion: 
-In a first step, the base model is used to generate (noisy) latents, 
-which are then further processed with a refinement model (available here: https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/) specialized for the final denoising steps.
-Note that the base model can be used as a standalone module.
+This model implements Conceptrol, a training-free method that boosts zero-shot personalized image generation across Stable Diffusion, SDXL, and FLUX. It works without additional training, data, or models.
 
-Alternatively, we can use a two-stage pipeline as follows: 
-First, the base model is used to generate latents of the desired output size. 
-In the second step, we use a specialized high-resolution model and apply a technique called SDEdit (https://arxiv.org/abs/2108.01073, also known as "img2img") 
-to the latents generated in the first step, using the same prompt. This technique is slightly slower than the first one, as it requires more function evaluations.
+<p align="center">
+  <img src="demo/teaser.png">
+</p>
 
-Source code is available at https://github.com/Stability-AI/generative-models .
+[Conceptrol: Concept Control of Zero-shot Personalized Image Generation](https://huggingface.co/papers/2503.06568)
 
-### Model Description
+**Abstract:**
 
-- **Developed by:** Stability AI
-- **Model type:** Diffusion-based text-to-image generative model
-- **License:** [CreativeML Open RAIL++-M License](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md)
-- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses two fixed, pretrained text encoders ([OpenCLIP-ViT/G](https://github.com/mlfoundations/open_clip) and [CLIP-ViT/L](https://github.com/openai/CLIP/tree/main)).
-- **Resources for more information:** Check out our [GitHub Repository](https://github.com/Stability-AI/generative-models) and the [SDXL report on arXiv](https://arxiv.org/abs/2307.01952).
+Personalized image generation with text-to-image diffusion models generates unseen images based on reference image content. Zero-shot adapter methods such as IP-Adapter and OminiControl are especially interesting because they do not require test-time fine-tuning. However, they struggle to balance preserving personalized content and adherence to the text prompt.  We identify a critical design flaw resulting in this performance gap: current adapters inadequately integrate personalization images with the textual descriptions. The generated images, therefore, replicate the personalized content rather than adhere to the text prompt instructions. Yet the base text-to-image has strong conceptual understanding capabilities that can be leveraged.
 
-### Model Sources
+We propose Conceptrol, a simple yet effective framework that enhances zero-shot adapters without adding computational overhead. Conceptrol constrains the attention of visual specification with a textual concept mask that improves subject-driven generation capabilities. It achieves as much as 89% improvement on personalization benchmarks over the vanilla IP-Adapter and can even outperform fine-tuning approaches such as Dreambooth LoRA.
 
-For research purposes, we recommend our `generative-models` Github repository (https://github.com/Stability-AI/generative-models), which implements the most popular diffusion frameworks (both training and inference) and for which new functionalities like distillation will be added over time.
-[Clipdrop](https://clipdrop.co/stable-diffusion) provides free SDXL inference.
+## Quick Start
 
-- **Repository:** https://github.com/Stability-AI/generative-models
-- **Demo:** https://clipdrop.co/stable-diffusion
+#### 1. Environment Setup
 
-
-## Evaluation
-![comparison](comparison.png)
-The chart above evaluates user preference for SDXL (with and without refinement) over SDXL 0.9 and Stable Diffusion 1.5 and 2.1. 
-The SDXL base model performs significantly better than the previous variants, and the model combined with the refinement module achieves the best overall performance.
-
-
-### 🧨 Diffusers 
-
-Make sure to upgrade diffusers to >= 0.19.0:
-```
-pip install diffusers --upgrade
-```
-
-In addition make sure to install `transformers`, `safetensors`, `accelerate` as well as the invisible watermark:
-```
-pip install invisible_watermark transformers accelerate safetensors
-```
-
-To just use the base model, you can run:
-
-```py
-from diffusers import DiffusionPipeline
-import torch
-
-pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, use_safetensors=True, variant="fp16")
-pipe.to("cuda")
-
-# if using torch < 2.0
-# pipe.enable_xformers_memory_efficient_attention()
-
-prompt = "An astronaut riding a green horse"
-
-images = pipe(prompt=prompt).images[0]
-```
-
-To use the whole base + refiner pipeline as an ensemble of experts you can run:
-
-```py
-from diffusers import DiffusionPipeline
-import torch
-
-# load both base & refiner
-base = DiffusionPipeline.from_pretrained(
-    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
-)
-base.to("cuda")
-refiner = DiffusionPipeline.from_pretrained(
-    "stabilityai/stable-diffusion-xl-refiner-1.0",
-    text_encoder_2=base.text_encoder_2,
-    vae=base.vae,
-    torch_dtype=torch.float16,
-    use_safetensors=True,
-    variant="fp16",
-)
-refiner.to("cuda")
-
-# Define how many steps and what % of steps to be run on each experts (80/20) here
-n_steps = 40
-high_noise_frac = 0.8
-
-prompt = "A majestic lion jumping from a big stone at night"
-
-# run both experts
-image = base(
-    prompt=prompt,
-    num_inference_steps=n_steps,
-    denoising_end=high_noise_frac,
-    output_type="latent",
-).images
-image = refiner(
-    prompt=prompt,
-    num_inference_steps=n_steps,
-    denoising_start=high_noise_frac,
-    image=image,
-).images[0]
-```
-
-When using `torch >= 2.0`, you can improve the inference speed by 20-30% with torch.compile. Simple wrap the unet with torch compile before running the pipeline:
-```py
-pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)
+``` bash
+conda create -n conceptrol python=3.10
+conda activate conceptrol
+pip install -r requirements.txt
 ```
 
-If you are limited by GPU VRAM, you can enable *cpu offloading* by calling `pipe.enable_model_cpu_offload`
-instead of `.to("cuda")`:
+#### 2. Go to `demo_sd.ipynb` / `demo_sdxl.ipynb` / `demo_flux.py` for fun!
 
-```diff
-- pipe.to("cuda")
-+ pipe.enable_model_cpu_offload()
-```
-
-For more information on how to use Stable Diffusion XL with `diffusers`, please have a look at [the Stable Diffusion XL Docs](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl).
-
-### Optimum
-[Optimum](https://github.com/huggingface/optimum) provides a Stable Diffusion pipeline compatible with both [OpenVINO](https://docs.openvino.ai/latest/index.html) and [ONNX Runtime](https://onnxruntime.ai/).
-
-#### OpenVINO
-
-To install Optimum with the dependencies required for OpenVINO :
+## Local Setup using Gradio
 
-```bash
-pip install optimum[openvino]
+#### 1. Start Gradio Interface
+``` bash
+pip install gradio
+gradio gradio_src/app.py
 ```
 
-To load an OpenVINO model and run inference with OpenVINO Runtime, you need to replace `StableDiffusionXLPipeline` with Optimum `OVStableDiffusionXLPipeline`. In case you want to load a PyTorch model and convert it to the OpenVINO format on-the-fly, you can set `export=True`.
+#### 2. Use the GUI!
 
-```diff
-- from diffusers import StableDiffusionXLPipeline
-+ from optimum.intel import OVStableDiffusionXLPipeline
+## Supporting Models
 
-model_id = "stabilityai/stable-diffusion-xl-base-1.0"
-- pipeline = StableDiffusionXLPipeline.from_pretrained(model_id)
-+ pipeline = OVStableDiffusionXLPipeline.from_pretrained(model_id)
-prompt = "A majestic lion jumping from a big stone at night"
-image = pipeline(prompt).images[0]
-```
-
-You can find more examples (such as static reshaping and model compilation) in optimum [documentation](https://huggingface.co/docs/optimum/main/en/intel/inference#stable-diffusion-xl).
+| Model Name            |  Link                                             |
+|-----------------------|-------------------------------------------------------------|
+| Stable Diffusion 1.5  | [stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5)   |
+| Realistic Vision V5.1 | [SG161222/Realistic_Vision_V5.1_noVAE](https://huggingface.co/SG161222/Realistic_Vision_V5.1_noVAE) |
+| Stable Diffusion XL-1024   | [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) |
+| Animagine XL v4.0 |   [cagliostrolab/animagine-xl-4.0](https://huggingface.co/cagliostrolab/animagine-xl-4.0)|
+| Realistic Vision XL V5.0 | [SG161222/RealVisXL_V5.0](https://huggingface.co/SG161222/RealVisXL_V5.0) |
+| FLUX-schnell | [black-forest-labs/FLUX.1-schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell) |
 
+| Adapter Name            |  Link                                             |
+|-----------------------|-------------------------------------------------------------|
+| IP-Adapter  | [h94/IP-Adapter](https://huggingface.co/h94/IP-Adapter/tree/main)  |
+| OminiControl | [Yuanshi/OminiControl](https://huggingface.co/Yuanshi/OminiControl) |
 
-#### ONNX
 
-To install Optimum with the dependencies required for ONNX Runtime inference :
+## Source Code
 
-```bash
-pip install optimum[onnxruntime]
-```
+https://github.com/QY-H00/Conceptrol
 
-To load an ONNX model and run inference with ONNX Runtime, you need to replace `StableDiffusionXLPipeline` with Optimum `ORTStableDiffusionXLPipeline`. In case you want to load a PyTorch model and convert it to the ONNX format on-the-fly, you can set `export=True`.
+## Citation
 
-```diff
-- from diffusers import StableDiffusionXLPipeline
-+ from optimum.onnxruntime import ORTStableDiffusionXLPipeline
-
-model_id = "stabilityai/stable-diffusion-xl-base-1.0"
-- pipeline = StableDiffusionXLPipeline.from_pretrained(model_id)
-+ pipeline = ORTStableDiffusionXLPipeline.from_pretrained(model_id)
-prompt = "A majestic lion jumping from a big stone at night"
-image = pipeline(prompt).images[0]
+``` bibtex
+@article{he2025conceptrol,
+  title={Conceptrol: Concept Control of Zero-shot Personalized Image Generation},
+  author={Qiyuan He and Angela Yao},
+  journal={arXiv preprint arXiv:2503.06568},
+  year={2025}
+}
 ```
 
-You can find more examples in optimum [documentation](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/models#stable-diffusion-xl).
-
-
-## Uses
-
-### Direct Use
-
-The model is intended for research purposes only. Possible research areas and tasks include
-
-- Generation of artworks and use in design and other artistic processes.
-- Applications in educational or creative tools.
-- Research on generative models.
-- Safe deployment of models which have the potential to generate harmful content.
-- Probing and understanding the limitations and biases of generative models.
-
-Excluded uses are described below.
-
-### Out-of-Scope Use
-
-The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
-
-## Limitations and Bias
-
-### Limitations
+## Acknowledgement
 
-- The model does not achieve perfect photorealism
-- The model cannot render legible text
-- The model struggles with more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
-- Faces and people in general may not be generated properly.
-- The autoencoding part of the model is lossy.
+We thank the following repositories for their great work: 
 
-### Bias
-While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
+[diffusers](https://github.com/huggingface/diffusers), 
+[transformers](https://github.com/huggingface/transformers), 
+[IP-Adapter](https://github.com/tencent-ailab/IP-Adapter), 
+[OminiControl](https://github.com/Yuanshi9815/OminiControl)