# BLIP-Diffusion

BLIP-Diffusion was proposed in [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://huggingface.co/papers/2305.14720). It enables zero-shot subject-driven generation and control-guided zero-shot generation.


The abstract from the paper is:

*Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties preserving the subject fidelity. To overcome these limitations, we introduce BLIP-Diffusion, a new subject-driven image generation model that supports multimodal control which consumes inputs of subject images and text prompts. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation. We first pre-train the multimodal encoder following BLIP-2 to produce visual representation aligned with the text. Then we design a subject representation learning task which enables a diffusion model to leverage such visual representation and generates new subject renditions. Compared with previous methods such as DreamBooth, our model enables zero-shot subject-driven generation, and efficient fine-tuning for customized subject with up to 20x speedup. We also demonstrate that BLIP-Diffusion can be flexibly combined with existing techniques such as ControlNet and prompt-to-prompt to enable novel subject-driven generation and editing applications. Project page at [this https URL](https://dxli94.github.io/BLIP-Diffusion-website/).*

The original codebase can be found at [salesforce/LAVIS](https://github.com/salesforce/LAVIS/tree/main/projects/blip-diffusion). You can find the official BLIP-Diffusion checkpoints under the [hf.co/SalesForce](https://hf.co/SalesForce) organization.

`BlipDiffusionPipeline` and `BlipDiffusionControlNetPipeline` were contributed by [`ayushtues`](https://github.com/ayushtues/).

> [!TIP]
> Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to efficiently load the same components into multiple pipelines.


## BlipDiffusionPipeline[[diffusers.BlipDiffusionPipeline]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.BlipDiffusionPipeline</name><anchor>diffusers.BlipDiffusionPipeline</anchor><source>https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/blip_diffusion/pipeline_blip_diffusion.py#L80</source><parameters>[{"name": "tokenizer", "val": ": CLIPTokenizer"}, {"name": "text_encoder", "val": ": ContextCLIPTextModel"}, {"name": "vae", "val": ": AutoencoderKL"}, {"name": "unet", "val": ": UNet2DConditionModel"}, {"name": "scheduler", "val": ": PNDMScheduler"}, {"name": "qformer", "val": ": Blip2QFormerModel"}, {"name": "image_processor", "val": ": BlipImageProcessor"}, {"name": "ctx_begin_pos", "val": ": int = 2"}, {"name": "mean", "val": ": typing.List[float] = None"}, {"name": "std", "val": ": typing.List[float] = None"}]</parameters><paramsdesc>- **tokenizer** (`CLIPTokenizer`) --
  Tokenizer for the text encoder
- **text_encoder** (`ContextCLIPTextModel`) --
  Text encoder to encode the text prompt
- **vae** ([AutoencoderKL](/docs/diffusers/main/en/api/models/autoencoderkl#diffusers.AutoencoderKL)) --
  VAE model to map the latents to the image
- **unet** ([UNet2DConditionModel](/docs/diffusers/main/en/api/models/unet2d-cond#diffusers.UNet2DConditionModel)) --
  Conditional U-Net architecture to denoise the image embedding.
- **scheduler** ([PNDMScheduler](/docs/diffusers/main/en/api/schedulers/pndm#diffusers.PNDMScheduler)) --
  A scheduler to be used in combination with `unet` to generate image latents.
- **qformer** (`Blip2QFormerModel`) --
  QFormer model to get multi-modal embeddings from the text and image.
- **image_processor** (`BlipImageProcessor`) --
  Image Processor to preprocess and postprocess the image.
- **ctx_begin_pos** (int, `optional`, defaults to 2) --
  Position of the context token in the text encoder.</paramsdesc><paramgroups>0</paramgroups></docstring>

Pipeline for Zero-Shot Subject Driven Generation using Blip Diffusion.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods the
library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>__call__</name><anchor>diffusers.BlipDiffusionPipeline.__call__</anchor><source>https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/blip_diffusion/pipeline_blip_diffusion.py#L192</source><parameters>[{"name": "prompt", "val": ": typing.List[str]"}, {"name": "reference_image", "val": ": Image"}, {"name": "source_subject_category", "val": ": typing.List[str]"}, {"name": "target_subject_category", "val": ": typing.List[str]"}, {"name": "latents", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "guidance_scale", "val": ": float = 7.5"}, {"name": "height", "val": ": int = 512"}, {"name": "width", "val": ": int = 512"}, {"name": "num_inference_steps", "val": ": int = 50"}, {"name": "generator", "val": ": typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None"}, {"name": "neg_prompt", "val": ": typing.Optional[str] = ''"}, {"name": "prompt_strength", "val": ": float = 1.0"}, {"name": "prompt_reps", "val": ": int = 20"}, {"name": "output_type", "val": ": typing.Optional[str] = 'pil'"}, {"name": "return_dict", "val": ": bool = True"}]</parameters><paramsdesc>- **prompt** (`List[str]`) --
  The prompt or prompts to guide the image generation.
- **reference_image** (`PIL.Image.Image`) --
  The reference image to condition the generation on.
- **source_subject_category** (`List[str]`) --
  The source subject category.
- **target_subject_category** (`List[str]`) --
  The target subject category.
- **latents** (`torch.Tensor`, *optional*) --
  Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
  generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
  tensor will be generated by random sampling.
- **guidance_scale** (`float`, *optional*, defaults to 7.5) --
  Guidance scale as defined in [Classifier-Free Diffusion
  Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
  of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting
  `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to
  the text `prompt`, usually at the expense of lower image quality.
- **height** (`int`, *optional*, defaults to 512) --
  The height of the generated image.
- **width** (`int`, *optional*, defaults to 512) --
  The width of the generated image.
- **num_inference_steps** (`int`, *optional*, defaults to 50) --
  The number of denoising steps. More denoising steps usually lead to a higher quality image at the
  expense of slower inference.
- **generator** (`torch.Generator` or `List[torch.Generator]`, *optional*) --
  One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
  to make generation deterministic.
- **neg_prompt** (`str`, *optional*, defaults to "") --
  The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored
  if `guidance_scale` is less than `1`).
- **prompt_strength** (`float`, *optional*, defaults to 1.0) --
  The strength of the prompt. Specifies the number of times the prompt is repeated along with prompt_reps
  to amplify the prompt.
- **prompt_reps** (`int`, *optional*, defaults to 20) --
  The number of times the prompt is repeated along with prompt_strength to amplify the prompt.
- **output_type** (`str`, *optional*, defaults to `"pil"`) --
  The output format of the generate image. Choose between: `"pil"` (`PIL.Image.Image`), `"np"`
  (`np.array`) or `"pt"` (`torch.Tensor`).
- **return_dict** (`bool`, *optional*, defaults to `True`) --
  Whether or not to return a [ImagePipelineOutput](/docs/diffusers/main/en/api/pipelines/dit#diffusers.ImagePipelineOutput) instead of a plain tuple.</paramsdesc><paramgroups>0</paramgroups><rettype>[ImagePipelineOutput](/docs/diffusers/main/en/api/pipelines/dit#diffusers.ImagePipelineOutput) or `tuple`</rettype></docstring>

Function invoked when calling the pipeline for generation.



<ExampleCodeBlock anchor="diffusers.BlipDiffusionPipeline.__call__.example">

Examples:
```py
>>> from diffusers.pipelines import BlipDiffusionPipeline
>>> from diffusers.utils import load_image
>>> import torch

>>> blip_diffusion_pipe = BlipDiffusionPipeline.from_pretrained(
...     "Salesforce/blipdiffusion", torch_dtype=torch.float16
... ).to("cuda")


>>> cond_subject = "dog"
>>> tgt_subject = "dog"
>>> text_prompt_input = "swimming underwater"

>>> cond_image = load_image(
...     "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/dog.jpg"
... )
>>> guidance_scale = 7.5
>>> num_inference_steps = 25
>>> negative_prompt = "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, out of frame, ugly, bad anatomy, bad proportions, deformed, blurry, duplicate"


>>> output = blip_diffusion_pipe(
...     text_prompt_input,
...     cond_image,
...     cond_subject,
...     tgt_subject,
...     guidance_scale=guidance_scale,
...     num_inference_steps=num_inference_steps,
...     neg_prompt=negative_prompt,
...     height=512,
...     width=512,
... ).images
>>> output[0].save("image.png")
```

</ExampleCodeBlock>







</div></div>

## BlipDiffusionControlNetPipeline[[diffusers.BlipDiffusionControlNetPipeline]]
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class diffusers.BlipDiffusionControlNetPipeline</name><anchor>diffusers.BlipDiffusionControlNetPipeline</anchor><source>https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/controlnet/pipeline_controlnet_blip_diffusion.py#L87</source><parameters>[{"name": "tokenizer", "val": ": CLIPTokenizer"}, {"name": "text_encoder", "val": ": ContextCLIPTextModel"}, {"name": "vae", "val": ": AutoencoderKL"}, {"name": "unet", "val": ": UNet2DConditionModel"}, {"name": "scheduler", "val": ": PNDMScheduler"}, {"name": "qformer", "val": ": Blip2QFormerModel"}, {"name": "controlnet", "val": ": ControlNetModel"}, {"name": "image_processor", "val": ": BlipImageProcessor"}, {"name": "ctx_begin_pos", "val": ": int = 2"}, {"name": "mean", "val": ": typing.List[float] = None"}, {"name": "std", "val": ": typing.List[float] = None"}]</parameters><paramsdesc>- **tokenizer** (`CLIPTokenizer`) --
  Tokenizer for the text encoder
- **text_encoder** (`ContextCLIPTextModel`) --
  Text encoder to encode the text prompt
- **vae** ([AutoencoderKL](/docs/diffusers/main/en/api/models/autoencoderkl#diffusers.AutoencoderKL)) --
  VAE model to map the latents to the image
- **unet** ([UNet2DConditionModel](/docs/diffusers/main/en/api/models/unet2d-cond#diffusers.UNet2DConditionModel)) --
  Conditional U-Net architecture to denoise the image embedding.
- **scheduler** ([PNDMScheduler](/docs/diffusers/main/en/api/schedulers/pndm#diffusers.PNDMScheduler)) --
  A scheduler to be used in combination with `unet` to generate image latents.
- **qformer** (`Blip2QFormerModel`) --
  QFormer model to get multi-modal embeddings from the text and image.
- **controlnet** ([ControlNetModel](/docs/diffusers/main/en/api/models/controlnet#diffusers.ControlNetModel)) --
  ControlNet model to get the conditioning image embedding.
- **image_processor** (`BlipImageProcessor`) --
  Image Processor to preprocess and postprocess the image.
- **ctx_begin_pos** (int, `optional`, defaults to 2) --
  Position of the context token in the text encoder.</paramsdesc><paramgroups>0</paramgroups></docstring>

Pipeline for Canny Edge based Controlled subject-driven generation using Blip Diffusion.

This model inherits from [DiffusionPipeline](/docs/diffusers/main/en/api/pipelines/overview#diffusers.DiffusionPipeline). Check the superclass documentation for the generic methods the
library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)





<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>__call__</name><anchor>diffusers.BlipDiffusionControlNetPipeline.__call__</anchor><source>https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/controlnet/pipeline_controlnet_blip_diffusion.py#L240</source><parameters>[{"name": "prompt", "val": ": typing.List[str]"}, {"name": "reference_image", "val": ": Image"}, {"name": "condtioning_image", "val": ": Image"}, {"name": "source_subject_category", "val": ": typing.List[str]"}, {"name": "target_subject_category", "val": ": typing.List[str]"}, {"name": "latents", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "guidance_scale", "val": ": float = 7.5"}, {"name": "height", "val": ": int = 512"}, {"name": "width", "val": ": int = 512"}, {"name": "num_inference_steps", "val": ": int = 50"}, {"name": "generator", "val": ": typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None"}, {"name": "neg_prompt", "val": ": typing.Optional[str] = ''"}, {"name": "prompt_strength", "val": ": float = 1.0"}, {"name": "prompt_reps", "val": ": int = 20"}, {"name": "output_type", "val": ": typing.Optional[str] = 'pil'"}, {"name": "return_dict", "val": ": bool = True"}]</parameters><paramsdesc>- **prompt** (`List[str]`) --
  The prompt or prompts to guide the image generation.
- **reference_image** (`PIL.Image.Image`) --
  The reference image to condition the generation on.
- **condtioning_image** (`PIL.Image.Image`) --
  The conditioning canny edge image to condition the generation on.
- **source_subject_category** (`List[str]`) --
  The source subject category.
- **target_subject_category** (`List[str]`) --
  The target subject category.
- **latents** (`torch.Tensor`, *optional*) --
  Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image
  generation. Can be used to tweak the same generation with different prompts. If not provided, a latents
  tensor will be generated by random sampling.
- **guidance_scale** (`float`, *optional*, defaults to 7.5) --
  Guidance scale as defined in [Classifier-Free Diffusion
  Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2.
  of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting
  `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to
  the text `prompt`, usually at the expense of lower image quality.
- **height** (`int`, *optional*, defaults to 512) --
  The height of the generated image.
- **width** (`int`, *optional*, defaults to 512) --
  The width of the generated image.
- **seed** (`int`, *optional*, defaults to 42) --
  The seed to use for random generation.
- **num_inference_steps** (`int`, *optional*, defaults to 50) --
  The number of denoising steps. More denoising steps usually lead to a higher quality image at the
  expense of slower inference.
- **generator** (`torch.Generator` or `List[torch.Generator]`, *optional*) --
  One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html)
  to make generation deterministic.
- **neg_prompt** (`str`, *optional*, defaults to "") --
  The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored
  if `guidance_scale` is less than `1`).
- **prompt_strength** (`float`, *optional*, defaults to 1.0) --
  The strength of the prompt. Specifies the number of times the prompt is repeated along with prompt_reps
  to amplify the prompt.
- **prompt_reps** (`int`, *optional*, defaults to 20) --
  The number of times the prompt is repeated along with prompt_strength to amplify the prompt.</paramsdesc><paramgroups>0</paramgroups><rettype>[ImagePipelineOutput](/docs/diffusers/main/en/api/pipelines/dit#diffusers.ImagePipelineOutput) or `tuple`</rettype></docstring>

Function invoked when calling the pipeline for generation.



<ExampleCodeBlock anchor="diffusers.BlipDiffusionControlNetPipeline.__call__.example">

Examples:
```py
>>> from diffusers.pipelines import BlipDiffusionControlNetPipeline
>>> from diffusers.utils import load_image
>>> from controlnet_aux import CannyDetector
>>> import torch

>>> blip_diffusion_pipe = BlipDiffusionControlNetPipeline.from_pretrained(
...     "Salesforce/blipdiffusion-controlnet", torch_dtype=torch.float16
... ).to("cuda")

>>> style_subject = "flower"
>>> tgt_subject = "teapot"
>>> text_prompt = "on a marble table"

>>> cldm_cond_image = load_image(
...     "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/kettle.jpg"
... ).resize((512, 512))
>>> canny = CannyDetector()
>>> cldm_cond_image = canny(cldm_cond_image, 30, 70, output_type="pil")
>>> style_image = load_image(
...     "https://huggingface.co/datasets/ayushtues/blipdiffusion_images/resolve/main/flower.jpg"
... )
>>> guidance_scale = 7.5
>>> num_inference_steps = 50
>>> negative_prompt = "over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, worst quality, low quality, jpeg artifacts, morbid, mutilated, out of frame, ugly, bad anatomy, bad proportions, deformed, blurry, duplicate"


>>> output = blip_diffusion_pipe(
...     text_prompt,
...     style_image,
...     cldm_cond_image,
...     style_subject,
...     tgt_subject,
...     guidance_scale=guidance_scale,
...     num_inference_steps=num_inference_steps,
...     neg_prompt=negative_prompt,
...     height=512,
...     width=512,
... ).images
>>> output[0].save("image.png")
```

</ExampleCodeBlock>







</div></div>

<EditOnGithub source="https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/blip_diffusion.md" />