#
ColPali

ColPali is a model designed to retrieve documents by analyzing their visual features. Unlike traditional systems that rely heavily on text extraction and OCR, ColPali treats each page as an image. It uses Paligemma-3B to capture not only text, but also the layout, tables, charts, and other visual elements to create detailed embeddings. This offers a more comprehensive understanding of documents and enables more efficient and accurate retrieval.

You can find all the original ColPali checkpoints under the ColPali collection.

Click on the ColPali models in the right sidebar for more examples of how to use ColPali for image retrieval.

image retrieval

    import requests
    import torch
    from PIL import Image
    from transformers import ColPaliForRetrieval, ColPaliProcessor

    # Load model (bfloat16 support is limited; fallback to float32 if needed)
    model = ColPaliForRetrieval.from_pretrained(
        "vidore/colpali-v1.2-hf",
        torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
        device_map="auto",  # "cpu", "cuda", or "mps" for Apple Silicon
    ).eval()

    processor = ColPaliProcessor.from_pretrained(model_name)

    url1 = "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg"
    url2 = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg"

    images = [
        Image.open(requests.get(url1, stream=True).raw),
        Image.open(requests.get(url2, stream=True).raw),
    ]

    queries = [
        "Who printed the edition of Romeo and Juliet?",
        "When was the United States Declaration of Independence proclaimed?",
    ]

    # Process the inputs
    inputs_images = processor(images=images, return_tensors="pt").to(model.device)
    inputs_text = processor(text=queries, return_tensors="pt").to(model.device)

    # Forward pass
    with torch.no_grad():
        image_embeddings = model(**inputs_images).embeddings
        query_embeddings = model(**inputs_text).embeddings

    scores = processor.score_retrieval(query_embeddings, image_embeddings)

    print("Retrieval scores (query x image):")
    print(scores)

Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the Quantization overview for more available quantization backends.

The example below uses bitsandbytes to quantize the weights to int4.

    import requests
    import torch
    from PIL import Image
    from transformers import ColPaliForRetrieval, ColPaliProcessor
    from transformers import BitsAndBytesConfig

    # 4-bit quantization configuration
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16,
    )

    model_name = "vidore/colpali-v1.2-hf"

    # Load model
    model = ColPaliForRetrieval.from_pretrained(
        model_name,
        quantization_config=bnb_config,
        device_map="cuda"
    ).eval()

    processor = ColPaliProcessor.from_pretrained(model_name)

    url1 = "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg"
    url2 = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg"

    images = [
        Image.open(requests.get(url1, stream=True).raw),
        Image.open(requests.get(url2, stream=True).raw),
    ]

    queries = [
        "Who printed the edition of Romeo and Juliet?",
        "When was the United States Declaration of Independence proclaimed?",
    ]

    # Process the inputs
    inputs_images = processor(images=images, return_tensors="pt").to(model.device)
    inputs_text = processor(text=queries, return_tensors="pt").to(model.device)

    # Forward pass
    with torch.no_grad():
        image_embeddings = model(**inputs_images).embeddings
        query_embeddings = model(**inputs_text).embeddings

    scores = processor.score_retrieval(query_embeddings, image_embeddings)

    print("Retrieval scores (query x image):")
    print(scores)

##
Notes

- score_retrieval() returns a 2D tensor where the first dimension is the number of queries and the second dimension is the number of images. A higher score indicates more similarity between the query and image.

##
ColPaliConfig

###
class transformers.ColPaliConfig

< source >

( vlm_config = None text_config = None embedding_dim: int = 128 **kwargs )

Parameters

- vlm_config (`PretrainedConfig`, optional) — Configuration of the VLM backbone model.
- text_config (`PretrainedConfig`, optional) — Configuration of the text backbone model. Overrides the `text_config` attribute of the `vlm_config` if provided.
- embedding_dim (`int`, optional, defaults to 128) — Dimension of the multi-vector embeddings produced by the model.

Configuration class to store the configuration of a ColPaliForRetrieval. It is used to instantiate an instance of `ColPaliForRetrieval` according to the specified arguments, defining the model architecture following the methodology from the “ColPali: Efficient Document Retrieval with Vision Language Models” paper.

Creating a configuration with the default settings will result in a configuration where the VLM backbone is set to the default PaliGemma configuration, i.e the one from vidore/colpali-v1.2.

The ColPali config is very similar to `PaligemmaConfig`, but with an extra attribute defining the embedding dimension.

Note that contrarily to what the class name suggests (actually the name refers to the ColPali methodology), you can use a different VLM backbone model than PaliGemma by passing the corresponding VLM configuration to the class constructor.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    from transformers.models.colpali import ColPaliConfig, ColPaliForRetrieval

    config = ColPaliConfig()
    model = ColPaliForRetrieval(config)

##
ColPaliProcessor

###
class transformers.ColPaliProcessor

< source >

( image_processor = None tokenizer = None chat_template = None **kwargs )

Parameters

- image_processor (SiglipImageProcessor, optional) — The image processor is a required input.
- tokenizer (LlamaTokenizerFast, optional) — The tokenizer is a required input.
- chat_template (`str`, optional) — A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string.

Constructs a ColPali processor which wraps a PaliGemmaProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score.

ColPaliProcessor offers all the functionalities of PaliGemmaProcessor. See the `__call__()` for more information.

####
batch_decode

< source >

( *args **kwargs )

This method forwards all its arguments to GemmaTokenizerFast’s batch_decode(). Please refer to the docstring of this method for more information.

####
decode

< source >

( *args **kwargs )

This method forwards all its arguments to GemmaTokenizerFast’s decode(). Please refer to the docstring of this method for more information.

####
process_images

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] = None **kwargs: typing_extensions.Unpack[transformers.models.colpali.processing_colpali.ColPaliProcessorKwargs] ) →
BatchFeature

Parameters

- images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`) — The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width.
- return_tensors (`str` or TensorType, optional) — If set, will return tensors of a particular framework. Acceptable values are:
  - `'tf'`: Return TensorFlow `tf.constant` objects.
  - `'pt'`: Return PyTorch `torch.Tensor` objects.
  - `'np'`: Return NumPy `np.ndarray` objects.
  - `'jax'`: Return JAX `jnp.ndarray` objects.

Returns

BatchFeature

A BatchFeature with the following fields:

- input_ids — List of token ids to be fed to a model.
- attention_mask — List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if “attention_mask” is in `self.model_input_names` and if `text` is not `None`).
- pixel_values — Pixel values to be fed to a model. Returned when `images` is not `None`.

Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor’s `ColPaliProcessor.__call__()`.

This method forwards the `images` and `kwargs` arguments to SiglipImageProcessor’s call().

####
process_queries

< source >

( text: typing.Union[str, typing.List[str]] **kwargs: typing_extensions.Unpack[transformers.models.colpali.processing_colpali.ColPaliProcessorKwargs] ) →
BatchFeature

Parameters

- text (`str`, `List[str]`, `List[List[str]]`) — The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).
- return_tensors (`str` or TensorType, optional) — If set, will return tensors of a particular framework. Acceptable values are:
  - `'tf'`: Return TensorFlow `tf.constant` objects.
  - `'pt'`: Return PyTorch `torch.Tensor` objects.
  - `'np'`: Return NumPy `np.ndarray` objects.
  - `'jax'`: Return JAX `jnp.ndarray` objects.

Returns

BatchFeature

A BatchFeature with the following fields:

- input_ids — List of token ids to be fed to a model.
- attention_mask — List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if “attention_mask” is in `self.model_input_names` and if `text` is not `None`).

Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor’s `ColPaliProcessor.__call__()`.

This method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast’s call().

####
score_retrieval

< source >

( query_embeddings: typing.Union[ForwardRef('torch.Tensor'), typing.List[ForwardRef('torch.Tensor')]] passage_embeddings: typing.Union[ForwardRef('torch.Tensor'), typing.List[ForwardRef('torch.Tensor')]] batch_size: int = 128 output_dtype: typing.Optional[ForwardRef('torch.dtype')] = None output_device: typing.Union[ForwardRef('torch.device'), str] = 'cpu' ) →
`torch.Tensor`

Parameters

- query_embeddings (`Union[torch.Tensor, List[torch.Tensor]`) — Query embeddings.
- passage_embeddings (`Union[torch.Tensor, List[torch.Tensor]`) — Passage embeddings.
- batch_size (`int`, optional, defaults to 128) — Batch size for computing scores.
- output_dtype (`torch.dtype`, optional, defaults to `torch.float32`) — The dtype of the output tensor. If `None`, the dtype of the input embeddings is used.
- output_device (`torch.device` or `str`, optional, defaults to “cpu”) — The device of the output tensor.

Returns

`torch.Tensor`

A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the “cpu” device.

Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page.

Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors.

##
ColPaliForRetrieval

###
class transformers.ColPaliForRetrieval

< source >

( config: ColPaliConfig )

In our proposed ColPali approach, we leverage VLMs to construct efficient multi-vector embeddings directly from document images (“screenshots”) for document retrieval. We train the model to maximize the similarity between these document embeddings and the corresponding query embeddings, using the late interaction method introduced in ColBERT.

Using ColPali removes the need for potentially complex and brittle layout recognition and OCR pipelines with a single model that can take into account both the textual and visual content (layout, charts, etc.) of a document.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None attention_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs ) →
`transformers.models.colpali.modeling_colpali.ColPaliForRetrievalOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. What are input IDs?
- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)) -- The tensors corresponding to the input images. Pixel values can be obtained using [AutoImageProcessor](/docs/transformers/main/en/model_doc/auto#transformers.AutoImageProcessor). See [SiglipImageProcessor.__call__()](/docs/transformers/main/en/model_doc/vit#transformers.ViTFeatureExtractor.__call__) for details ([]`PaliGemmaProcessor`] uses SiglipImageProcessor for processing images). If none, ColPali will only process text (query embeddings).
- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked. What are attention masks? Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read `modeling_opt._prepare_decoder_attention_mask` and modify to your needs. See diagram 1 in the paper for more information on the default strategy.
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- kwargs (`Dict[str, Any]`, optional) — Additional key word arguments passed along to the vlm backbone model.

Returns

`transformers.models.colpali.modeling_colpali.ColPaliForRetrievalOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.colpali.modeling_colpali.ColPaliForRetrievalOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (ColPaliConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Language modeling loss (for next-token prediction).

- embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — The embeddings of the model.

- past_key_values (`tuple(tuple(torch.FloatTensor))`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`)

  Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- image_hidden_states (`torch.FloatTensor`, optional) — A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state.

The ColPaliForRetrieval forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

< > Update on GitHub