#
BridgeTower

##
Overview

BridgeTower モデルは、Xiao Xu、Chenfei Wu、Shachar Rosenman、Vasudev Lal、Wanxiang Che、Nan Duan BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning で提案されました。ドゥアン。このモデルの目標は、 各ユニモーダル エンコーダとクロスモーダル エンコーダの間のブリッジにより、クロスモーダル エンコーダの各層での包括的かつ詳細な対話が可能になり、追加のパフォーマンスと計算コストがほとんど無視できる程度で、さまざまな下流タスクで優れたパフォーマンスを実現します。

この論文は AAAI’23 会議に採択されました。

論文の要約は次のとおりです。

TWO-TOWER アーキテクチャを備えたビジョン言語 (VL) モデルは、近年の視覚言語表現学習の主流となっています。 現在の VL モデルは、軽量のユニモーダル エンコーダーを使用して、ディープ クロスモーダル エンコーダーで両方のモダリティを同時に抽出、位置合わせ、融合することを学習するか、事前にトレーニングされたディープ ユニモーダル エンコーダーから最終層のユニモーダル表現を上部のクロスモーダルエンコーダー。 どちらのアプローチも、視覚言語表現の学習を制限し、モデルのパフォーマンスを制限する可能性があります。この論文では、ユニモーダル エンコーダの最上位層とクロスモーダル エンコーダの各層の間の接続を構築する複数のブリッジ層を導入する BRIDGETOWER を提案します。 これにより、効果的なボトムアップのクロスモーダル調整と、クロスモーダル エンコーダー内の事前トレーニング済みユニモーダル エンコーダーのさまざまなセマンティック レベルの視覚表現とテキスト表現の間の融合が可能になります。 BRIDGETOWER は 4M 画像のみで事前トレーニングされており、さまざまな下流の視覚言語タスクで最先端のパフォーマンスを実現します。 特に、VQAv2 テスト標準セットでは、BRIDGETOWER は 78.73% の精度を達成し、同じ事前トレーニング データとほぼ無視できる追加パラメータと計算コストで以前の最先端モデル METER を 1.09% 上回りました。 特に、モデルをさらにスケーリングすると、BRIDGETOWER は 81.15% の精度を達成し、桁違いに大きなデータセットで事前トレーニングされたモデルを上回りました。

ブリッジタワー アーキテクチャ。 元の論文から抜粋。

このモデルは、Anahita Bhiwandiwalla、Tiep Le、Shaoyen Tseng 。元のコードは ここ にあります。

##
Usage tips and examples

BridgeTower は、ビジュアル エンコーダー、テキスト エンコーダー、および複数の軽量ブリッジ レイヤーを備えたクロスモーダル エンコーダーで構成されます。 このアプローチの目標は、各ユニモーダル エンコーダーとクロスモーダル エンコーダーの間にブリッジを構築し、クロスモーダル エンコーダーの各層で包括的かつ詳細な対話を可能にすることでした。 原則として、提案されたアーキテクチャでは、任意のビジュアル、テキスト、またはクロスモーダル エンコーダを適用できます。

BridgeTowerProcessor は、`RobertaTokenizer` と BridgeTowerImageProcessor を単一のインスタンスにラップし、両方の機能を実現します。 テキストをエンコードし、画像をそれぞれ用意します。

次の例は、BridgeTowerProcessor と BridgeTowerForContrastiveLearning を使用して対照学習を実行する方法を示しています。

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning
    >>> import requests
    >>> from PIL import Image

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")
    >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")

    >>> # forward pass
    >>> scores = dict()
    >>> for text in texts:
    ...     # prepare inputs
    ...     encoding = processor(image, text, return_tensors="pt")
    ...     outputs = model(**encoding)
    ...     scores[text] = outputs

次の例は、BridgeTowerProcessor と BridgeTowerForImageAndTextRetrieval を使用して画像テキストの取得を実行する方法を示しています。

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
    >>> import requests
    >>> from PIL import Image

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
    >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

    >>> # forward pass
    >>> scores = dict()
    >>> for text in texts:
    ...     # prepare inputs
    ...     encoding = processor(image, text, return_tensors="pt")
    ...     outputs = model(**encoding)
    ...     scores[text] = outputs.logits[0, 1].item()

次の例は、BridgeTowerProcessor と BridgeTowerForMaskedLM を使用してマスクされた言語モデリングを実行する方法を示しています。

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM
    >>> from PIL import Image
    >>> import requests

    >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
    >>> text = "a <mask> looking out of the window"

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
    >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

    >>> # prepare inputs
    >>> encoding = processor(image, text, return_tensors="pt")

    >>> # forward pass
    >>> outputs = model(**encoding)

    >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())

    >>> print(results)
    .a cat looking out of the window.

チップ：

- BridgeTower のこの実装では、`RobertaTokenizer` を使用してテキスト埋め込みを生成し、OpenAI の CLIP/ViT モデルを使用して視覚的埋め込みを計算します。
- 事前トレーニングされた bridgeTower-base および bridgetower マスクされた言語モデリングと画像テキスト マッチング のチェックポイント がリリースされました。
- 画像検索およびその他の下流タスクにおける BridgeTower のパフォーマンスについては、表 5 を参照してください。
- このモデルの PyTorch バージョンは、torch 1.10 以降でのみ使用できます。

##
BridgeTowerConfig

###
class transformers.BridgeTowerConfig

< source >

( share_cross_modal_transformer_layers = True hidden_act = 'gelu' hidden_size = 768 initializer_factor = 1 layer_norm_eps = 1e-05 share_link_tower_layers = False link_tower_type = 'add' num_attention_heads = 12 num_hidden_layers = 6 tie_word_embeddings = False init_layernorm_from_vision_encoder = False text_config = None vision_config = None **kwargs )

Parameters

- share_cross_modal_transformer_layers (`bool`, optional, defaults to `True`) — Whether cross modal transformer layers are shared.
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler.
- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- initializer_factor (`float`, optional, defaults to 1) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- layer_norm_eps (`float`, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
- share_link_tower_layers (`bool`, optional, defaults to `False`) — Whether the bride/link tower layers are shared.
- link_tower_type (`str`, optional, defaults to `"add"`) — Type of the bridge/link layer.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- num_hidden_layers (`int`, optional, defaults to 6) — Number of hidden layers in the Transformer encoder.
- tie_word_embeddings (`bool`, optional, defaults to `False`) — Whether to tie input and output embeddings.
- init_layernorm_from_vision_encoder (`bool`, optional, defaults to `False`) — Whether to init LayerNorm from the vision encoder.
- text_config (`dict`, optional) — Dictionary of configuration options used to initialize BridgeTowerTextConfig.
- vision_config (`dict`, optional) — Dictionary of configuration options used to initialize BridgeTowerVisionConfig.

This is the configuration class to store the configuration of a BridgeTowerModel. It is used to instantiate a BridgeTower model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base BridgeTower/bridgetower-base architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import BridgeTowerModel, BridgeTowerConfig

    >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration
    >>> configuration = BridgeTowerConfig()

    >>> # Initializing a model from the BridgeTower/bridgetower-base style configuration
    >>> model = BridgeTowerModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

####
from_text_vision_configs

< source >

( text_config: BridgeTowerTextConfig vision_config: BridgeTowerVisionConfig **kwargs )

Instantiate a BridgeTowerConfig (or a derived class) from BridgeTower text model configuration. Returns: BridgeTowerConfig: An instance of a configuration object

##
BridgeTowerTextConfig

###
class transformers.BridgeTowerTextConfig

< source >

( vocab_size = 50265 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 initializer_factor = 1 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 514 type_vocab_size = 1 layer_norm_eps = 1e-05 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True **kwargs )

Parameters

- vocab_size (`int`, optional, defaults to 50265) — Vocabulary size of the text part of the model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling BridgeTowerModel.
- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- intermediate_size (`int`, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
- hidden_act (`str` or `Callable`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"silu"` and `"gelu_new"` are supported.
- hidden_dropout_prob (`float`, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- attention_probs_dropout_prob (`float`, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
- max_position_embeddings (`int`, optional, defaults to 514) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
- type_vocab_size (`int`, optional, defaults to 2) — The vocabulary size of the `token_type_ids`.
- initializer_factor (`float`, optional, defaults to 1) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- layer_norm_eps (`float`, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
- position_embedding_type (`str`, optional, defaults to `"absolute"`) — Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to Self-Attention with Relative Position Representations (Shaw et al.). For more information on `"relative_key_query"`, please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.).
- is_decoder (`bool`, optional, defaults to `False`) — Whether the model is used as a decoder or not. If `False`, the model is used as an encoder.
- use_cache (`bool`, optional, defaults to `True`) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`.

This is the configuration class to store the text configuration of a BridgeTowerModel. The default values here are copied from RoBERTa. Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base BridegTower/bridgetower-base architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import BridgeTowerTextConfig

    >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration for the text model
    >>> configuration = BridgeTowerTextConfig()

    >>> # Accessing the configuration
    >>> configuration

##
BridgeTowerVisionConfig

###
class transformers.BridgeTowerVisionConfig

< source >

( hidden_size = 768 num_hidden_layers = 12 num_channels = 3 patch_size = 16 image_size = 288 initializer_factor = 1 layer_norm_eps = 1e-05 stop_gradient = False share_layernorm = True remove_last_layer = False **kwargs )

Parameters

- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in visual encoder model.
- patch_size (`int`, optional, defaults to 16) — The size (resolution) of each patch.
- image_size (`int`, optional, defaults to 288) — The size (resolution) of each image.
- initializer_factor (`float`, optional, defaults to 1) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- layer_norm_eps (`float`, optional, defaults to 1e-05) — The epsilon used by the layer normalization layers.
- stop_gradient (`bool`, optional, defaults to `False`) — Whether to stop gradient for training.
- share_layernorm (`bool`, optional, defaults to `True`) — Whether LayerNorm layers are shared.
- remove_last_layer (`bool`, optional, defaults to `False`) — Whether to remove the last layer from the vision encoder.

This is the configuration class to store the vision configuration of a BridgeTowerModel. Instantiating a configuration with the defaults will yield a similar configuration to that of the bridgetower-base BridgeTower/bridgetower-base architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import BridgeTowerVisionConfig

    >>> # Initializing a BridgeTower BridgeTower/bridgetower-base style configuration for the vision model
    >>> configuration = BridgeTowerVisionConfig()

    >>> # Accessing the configuration
    >>> configuration

##
BridgeTowerImageProcessor

###
class transformers.BridgeTowerImageProcessor

< source >

( do_resize: bool = True size: typing.Dict[str, int] = None size_divisor: int = 32 resample: Resampling = <Resampling.BICUBIC: 3> do_rescale: bool = True rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_center_crop: bool = True crop_size: typing.Dict[str, int] = None do_pad: bool = True **kwargs )

Parameters

- do_resize (`bool`, optional, defaults to `True`) — Whether to resize the image’s (height, width) dimensions to the specified `size`. Can be overridden by the `do_resize` parameter in the `preprocess` method.
- size (`Dict[str, int]` optional, defaults to `{'shortest_edge' -- 288}`): Resize the shorter side of the input to `size["shortest_edge"]`. The longer side will be limited to under `int((1333 / 800) * size["shortest_edge"])` while preserving the aspect ratio. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `size` parameter in the `preprocess` method.
- size_divisor (`int`, optional, defaults to 32) — The size by which to make sure both the height and width can be divided. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `size_divisor` parameter in the `preprocess` method.
- resample (`PILImageResampling`, optional, defaults to `Resampling.BICUBIC`) — Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `resample` parameter in the `preprocess` method.
- do_rescale (`bool`, optional, defaults to `True`) — Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method.
- rescale_factor (`int` or `float`, optional, defaults to `1/255`) — Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be overridden by the `rescale_factor` parameter in the `preprocess` method.
- do_normalize (`bool`, optional, defaults to `True`) — Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
- image_mean (`float` or `List[float]`, optional, defaults to `IMAGENET_STANDARD_MEAN`) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be overridden by the `image_mean` parameter in the `preprocess` method.
- image_std (`float` or `List[float]`, optional, defaults to `IMAGENET_STANDARD_STD`) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method.
- do_center_crop (`bool`, optional, defaults to `True`) — Whether to center crop the image. Can be overridden by the `do_center_crop` parameter in the `preprocess` method.
- crop_size (`Dict[str, int]`, optional) — Desired output size when applying center-cropping. Only has an effect if `do_center_crop` is set to `True`. Can be overridden by the `crop_size` parameter in the `preprocess` method. If unset defaults to `size`,
- do_pad (`bool`, optional, defaults to `True`) — Whether to pad the image to the `(max_height, max_width)` of the images in the batch. Can be overridden by the `do_pad` parameter in the `preprocess` method.

Constructs a BridgeTower image processor.

####
preprocess

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] do_resize: typing.Optional[bool] = None size: typing.Optional[typing.Dict[str, int]] = None size_divisor: typing.Optional[int] = None resample: Resampling = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Optional[float] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None do_pad: typing.Optional[bool] = None do_center_crop: typing.Optional[bool] = None crop_size: typing.Dict[str, int] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None )

Parameters

- images (`ImageInput`) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`.
- do_resize (`bool`, optional, defaults to `self.do_resize`) — Whether to resize the image.
- size (`Dict[str, int]`, optional, defaults to `self.size`) — Controls the size of the image after `resize`. The shortest edge of the image is resized to `size["shortest_edge"]` whilst preserving the aspect ratio. If the longest edge of this resized image is > `int(size["shortest_edge"] * (1333 / 800))`, then the image is resized again to make the longest edge equal to `int(size["shortest_edge"] * (1333 / 800))`.
- size_divisor (`int`, optional, defaults to `self.size_divisor`) — The image is resized to a size that is a multiple of this value.
- resample (`PILImageResampling`, optional, defaults to `self.resample`) — Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`.
- do_rescale (`bool`, optional, defaults to `self.do_rescale`) — Whether to rescale the image values between [0 - 1].
- rescale_factor (`float`, optional, defaults to `self.rescale_factor`) — Rescale factor to rescale the image by if `do_rescale` is set to `True`.
- do_normalize (`bool`, optional, defaults to `self.do_normalize`) — Whether to normalize the image.
- image_mean (`float` or `List[float]`, optional, defaults to `self.image_mean`) — Image mean to normalize the image by if `do_normalize` is set to `True`.
- image_std (`float` or `List[float]`, optional, defaults to `self.image_std`) — Image standard deviation to normalize the image by if `do_normalize` is set to `True`.
- do_pad (`bool`, optional, defaults to `self.do_pad`) — Whether to pad the image to the (max_height, max_width) in the batch. If `True`, a pixel mask is also created and returned.
- do_center_crop (`bool`, optional, defaults to `self.do_center_crop`) — Whether to center crop the image. If the input size is smaller than `crop_size` along any edge, the image is padded with 0’s and then center cropped.
- crop_size (`Dict[str, int]`, optional, defaults to `self.crop_size`) — Size of the image after center crop. If one edge the image is smaller than `crop_size`, it will be padded with zeros and then cropped
- return_tensors (`str` or `TensorType`, optional) — The type of tensors to return. Can be one of:
  - Unset: Return a list of `np.ndarray`.
  - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
  - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
  - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
  - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
- data_format (`ChannelDimension` or `str`, optional, defaults to `ChannelDimension.FIRST`) — The channel dimension format for the output image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - Unset: Use the channel dimension format of the input image.
- input_data_format (`ChannelDimension` or `str`, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

Preprocess an image or batch of images.

##
BridgeTowerProcessor

###
class transformers.BridgeTowerProcessor

< source >

( image_processor tokenizer )

Parameters

- image_processor (`BridgeTowerImageProcessor`) — An instance of BridgeTowerImageProcessor. The image processor is a required input.
- tokenizer (`RobertaTokenizerFast`) — An instance of [‘RobertaTokenizerFast`]. The tokenizer is a required input.

Constructs a BridgeTower processor which wraps a Roberta tokenizer and BridgeTower image processor into a single processor.

BridgeTowerProcessor offers all the functionalities of BridgeTowerImageProcessor and `RobertaTokenizerFast`. See the docstring of call() and `decode()` for more information.

####
__call__

< source >

( images text: typing.Union[str, typing.List[str], typing.List[typing.List[str]]] = None audio = None videos = None **kwargs: typing_extensions.Unpack[transformers.models.bridgetower.processing_bridgetower.BridgeTowerProcessorKwargs] )

This method uses BridgeTowerImageProcessor.call() method to prepare image(s) for the model, and RobertaTokenizerFast.call() to prepare text for the model.

Please refer to the docstring of the above two methods for more information.

##
BridgeTowerModel

###
class transformers.BridgeTowerModel

< source >

( config )

Parameters

- config (BridgeTowerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on top. This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None pixel_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None image_embeds: typing.Optional[torch.FloatTensor] = None image_token_type_idx: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None interpolate_pos_encoding: bool = False ) →
`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `({0})`) — Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. What are input IDs?
- attention_mask (`torch.FloatTensor` of shape `({0})`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked. What are attention masks?
- token_type_ids (`torch.LongTensor` of shape `({0})`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. What are token type IDs?
- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using BridgeTowerImageProcessor. See BridgeTowerImageProcessor.call() for details.
- pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`:
  - 1 for pixels that are real (i.e. not masked),
  - 0 for pixels that are padding (i.e. masked). `What are attention masks? <../glossary.html#attention-mask>`__
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, optional) — Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings.
- image_token_type_idx (`int`, optional) —
  - The token type ids for images.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- output_hidden_states (`bool`, optional) — If set to `True`, hidden states are returned as a list containing the hidden states of text, image, and cross-modal components respectively. i.e. `(hidden_states_text, hidden_states_image, hidden_states_cross_modal)` where each element is a list of the hidden states of the corresponding modality. `hidden_states_txt/img` are a list of tensors corresponding to unimodal hidden states and `hidden_states_cross_modal` is a list of tuples containing `cross_modal_text_hidden_states` and `cross_modal_image_hidden_states` of each brdige layer.
- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels are currently not supported.

Returns

`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerModelOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (BridgeTowerConfig) and inputs.

- text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`) — Sequence of hidden-states at the text output of the last layer of the model.

- image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`) — Sequence of hidden-states at the image output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`) — Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The BridgeTowerModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import BridgeTowerProcessor, BridgeTowerModel
    >>> from PIL import Image
    >>> import requests

    >>> # prepare image and text
    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> text = "hello world"
    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base")
    >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base")

    >>> inputs = processor(image, text, return_tensors="pt")
    >>> outputs = model(**inputs)
    >>> outputs.keys()
    odict_keys(['text_features', 'image_features', 'pooler_output'])

##
BridgeTowerForContrastiveLearning

###
class transformers.BridgeTowerForContrastiveLearning

< source >

( config )

Parameters

- config (BridgeTowerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None pixel_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None image_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = True return_dict: typing.Optional[bool] = None return_loss: typing.Optional[bool] = None ) →
`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `({0})`) — Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. What are input IDs?
- attention_mask (`torch.FloatTensor` of shape `({0})`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked. What are attention masks?
- token_type_ids (`torch.LongTensor` of shape `({0})`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. What are token type IDs?
- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using BridgeTowerImageProcessor. See BridgeTowerImageProcessor.call() for details.
- pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`:
  - 1 for pixels that are real (i.e. not masked),
  - 0 for pixels that are padding (i.e. masked). `What are attention masks? <../glossary.html#attention-mask>`__
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, optional) — Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings.
- image_token_type_idx (`int`, optional) —
  - The token type ids for images.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- return_loss (`bool`, optional) — Whether or not to return the contrastive loss.

Returns

`transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.bridgetower.modeling_bridgetower.BridgeTowerContrastiveOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (BridgeTowerConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `return_loss` is `True` — Image-text contrastive loss.
- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
- text_embeds (`torch.FloatTensor)`, optional, returned when model is initialized with `with_projection=True`) — The text embeddings obtained by applying the projection layer to the pooler_output.
- image_embeds (`torch.FloatTensor)`, optional, returned when model is initialized with `with_projection=True`) — The image embeddings obtained by applying the projection layer to the pooler_output.
- cross_embeds (`torch.FloatTensor)`, optional, returned when model is initialized with `with_projection=True`) — The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output.
- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

The BridgeTowerForContrastiveLearning forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning
    >>> import requests
    >>> from PIL import Image
    >>> import torch

    >>> image_urls = [
    ...     "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg",
    ...     "http://images.cocodataset.org/val2017/000000039769.jpg",
    ... ]
    >>> texts = ["two dogs in a car", "two cats sleeping on a couch"]
    >>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")
    >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc")

    >>> inputs = processor(images, texts, padding=True, return_tensors="pt")
    >>> loss = model(**inputs, return_loss=True).loss

    >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt")
    >>> loss_swapped = model(**inputs, return_loss=True).loss

    >>> print("Loss", round(loss.item(), 4))
    Loss 0.0019

    >>> print("Loss with swapped images", round(loss_swapped.item(), 4))
    Loss with swapped images 2.126

##
BridgeTowerForMaskedLM

###
class transformers.BridgeTowerForMaskedLM

< source >

( config )

Parameters

- config (BridgeTowerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

BridgeTower Model with a language modeling head on top as done during pretraining.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None pixel_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None image_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None ) →
transformers.modeling_outputs.MaskedLMOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. What are input IDs?
- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked. What are attention masks?
- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. What are token type IDs?
- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using BridgeTowerImageProcessor. See BridgeTowerImageProcessor.call() for details.
- pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`:
  - 1 for pixels that are real (i.e. not masked),
  - 0 for pixels that are padding (i.e. masked). `What are attention masks? <../glossary.html#attention-mask>`__
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, optional) — Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings.
- image_token_type_idx (`int`, optional) —
  - The token type ids for images.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`

Returns

transformers.modeling_outputs.MaskedLMOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.MaskedLMOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (BridgeTowerConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Masked language modeling (MLM) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The BridgeTowerForMaskedLM forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM
    >>> from PIL import Image
    >>> import requests

    >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
    >>> text = "a <mask> looking out of the window"

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
    >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

    >>> # prepare inputs
    >>> encoding = processor(image, text, return_tensors="pt")

    >>> # forward pass
    >>> outputs = model(**encoding)

    >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())

    >>> print(results)
    .a cat looking out of the window.

##
BridgeTowerForImageAndTextRetrieval

###
class transformers.BridgeTowerForImageAndTextRetrieval

< source >

( config )

Parameters

- config (BridgeTowerConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching.

This model is a PyTorch `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None pixel_values: typing.Optional[torch.FloatTensor] = None pixel_mask: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None image_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None labels: typing.Optional[torch.LongTensor] = None ) →
transformers.modeling_outputs.SequenceClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `({0})`) — Indices of input sequence tokens in the vocabulary. Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details. What are input IDs?
- attention_mask (`torch.FloatTensor` of shape `({0})`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked. What are attention masks?
- token_type_ids (`torch.LongTensor` of shape `({0})`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. What are token type IDs?
- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using BridgeTowerImageProcessor. See BridgeTowerImageProcessor.call() for details.
- pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, optional) — Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`:
  - 1 for pixels that are real (i.e. not masked),
  - 0 for pixels that are padding (i.e. masked). `What are attention masks? <../glossary.html#attention-mask>`__
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, optional) — Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings.
- image_token_type_idx (`int`, optional) —
  - The token type ids for images.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- labels (`torch.LongTensor` of shape `(batch_size, 1)`, optional) — Labels for computing the image-text matching loss. 0 means the pairs don’t match and 1 means they match. The pairs with 0 will be skipped for calculation.

Returns

transformers.modeling_outputs.SequenceClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (BridgeTowerConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The BridgeTowerForImageAndTextRetrieval forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
    >>> import requests
    >>> from PIL import Image

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)
    >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]

    >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")
    >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm")

    >>> # forward pass
    >>> scores = dict()
    >>> for text in texts:
    ...     # prepare inputs
    ...     encoding = processor(image, text, return_tensors="pt")
    ...     outputs = model(**encoding)
    ...     scores[text] = outputs.logits[0, 1].item()

< > Update on GitHub