#
DeiT

##
Overview

DeiT モデルは、Hugo Touvron、Matthieu Cord、Matthijs Douze、Francisco Massa、Alexandre Sablayrolles, Hervé Jégou.によって Training data-efficient image Transformers & distillation through attention で提案されました。 サブレイロール、エルヴェ・ジェグー。 Dosovitskiy et al., 2020 で紹介された Vision Transformer (ViT) は、既存の畳み込みニューラルと同等、またはそれを上回るパフォーマンスを発揮できることを示しました。 Transformer エンコーダ (BERT のような) を使用したネットワーク。ただし、その論文で紹介された ViT モデルには、次のトレーニングが必要でした。 外部データを使用して、数週間にわたる高価なインフラストラクチャ。 DeiT (データ効率の高い画像変換器) はさらに優れています 画像分類用に効率的にトレーニングされたトランスフォーマーにより、必要なデータとコンピューティング リソースがはるかに少なくなります。 オリジナルの ViT モデルとの比較。

論文の要約は次のとおりです。

最近、純粋に注意に基づくニューラル ネットワークが、画像などの画像理解タスクに対処できることが示されました。 分類。ただし、これらのビジュアル トランスフォーマーは、 インフラストラクチャが高価であるため、その採用が制限されています。この作業では、コンボリューションフリーの競争力のあるゲームを作成します。 Imagenet のみでトレーニングしてトランスフォーマーを作成します。 1 台のコンピューターで 3 日以内にトレーニングを行います。私たちの基準となるビジョン トランス (86M パラメータ) は、外部なしで ImageNet 上で 83.1% (単一クロップ評価) のトップ 1 の精度を達成します。 データ。さらに重要なのは、トランスフォーマーに特有の教師と生徒の戦略を導入することです。蒸留に依存している 学生が注意を払って教師から学ぶことを保証するトークン。私たちはこのトークンベースに興味を示します 特に convnet を教師として使用する場合。これにより、convnet と競合する結果を報告できるようになります。 Imagenet (最大 85.2% の精度が得られます) と他のタスクに転送するときの両方で。私たちはコードを共有し、 モデル。

このモデルは、nielsr によって提供されました。このモデルの TensorFlow バージョンは、amyeroberts によって追加されました。

##
Usage tips

- ViT と比較して、DeiT モデルはいわゆる蒸留トークンを使用して教師から効果的に学習します (これは、 DeiT 論文は、ResNet のようなモデルです)。蒸留トークンは、バックプロパゲーションを通じて、と対話することによって学習されます。 セルフアテンション層を介したクラス ([CLS]) とパッチ トークン。
- 抽出されたモデルを微調整するには 2 つの方法があります。(1) 上部に予測ヘッドを配置するだけの古典的な方法。 クラス トークンの最終的な非表示状態を抽出し、蒸留シグナルを使用しない、または (2) 両方の 予測ヘッドはクラス トークンの上と蒸留トークンの上にあります。その場合、[CLS] 予測は head は、head の予測とグラウンド トゥルース ラベル間の通常のクロスエントロピーを使用してトレーニングされます。 蒸留予測ヘッドは、硬蒸留 (予測と予測の間のクロスエントロピー) を使用してトレーニングされます。 蒸留ヘッドと教師が予測したラベル）。推論時に、平均予測を取得します。 最終的な予測として両頭の間で。 (2) は「蒸留による微調整」とも呼ばれます。 下流のデータセットですでに微調整されている教師。モデル的には (1) に相当します。 DeiTForImageClassification と (2) に対応します。 DeiTForImageClassificationWithTeacher。
- 著者らは (2) についてもソフト蒸留を試みたことに注意してください (この場合、蒸留予測ヘッドは 教師のソフトマックス出力に一致するように KL ダイバージェンスを使用してトレーニングしました）が、ハード蒸留が最良の結果をもたらしました。
- リリースされたすべてのチェックポイントは、ImageNet-1k のみで事前トレーニングおよび微調整されました。外部データは使用されませんでした。これは JFT-300M データセット/Imagenet-21k などの外部データを使用した元の ViT モデルとは対照的です。 事前トレーニング。
- DeiT の作者は、より効率的にトレーニングされた ViT モデルもリリースしました。これは、直接プラグインできます。 `ViTModel` または `ViTForImageClassification`。データなどのテクニック はるかに大規模なデータセットでのトレーニングをシミュレートするために、拡張、最適化、正則化が使用されました。 (ただし、事前トレーニングには ImageNet-1k のみを使用します)。 4 つのバリエーション (3 つの異なるサイズ) が利用可能です。 facebook/deit-tiny-patch16-224、facebook/deit-small-patch16-224、facebook/deit-base-patch16-224 および facebook/deit-base-patch16-384。以下を行うには DeiTImageProcessor を使用する必要があることに注意してください。 モデル用の画像を準備します。

##
Resources

DeiT を始めるのに役立つ公式 Hugging Face およびコミュニティ (🌎 で示されている) リソースのリスト。

Image Classification

- DeiTForImageClassification は、この サンプル スクリプト および ノートブック。
- 参照: 画像分類タスク ガイド

それに加えて:

- DeiTForMaskedImageModeling は、この サンプル スクリプト でサポートされています。

ここに含めるリソースの送信に興味がある場合は、お気軽にプル リクエストを開いてください。審査させていただきます。リソースは、既存のリソースを複製するのではなく、何か新しいものを示すことが理想的です。

##
DeiTConfig

###
class transformers.DeiTConfig

< source >

( hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 image_size = 224 patch_size = 16 num_channels = 3 qkv_bias = True encoder_stride = 16 pooler_output_size = None pooler_act = 'tanh' **kwargs )

Parameters

- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- intermediate_size (`int`, optional, defaults to 3072) — Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
- hidden_dropout_prob (`float`, optional, defaults to 0.0) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- attention_probs_dropout_prob (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- layer_norm_eps (`float`, optional, defaults to 1e-12) — The epsilon used by the layer normalization layers.
- image_size (`int`, optional, defaults to 224) — The size (resolution) of each image.
- patch_size (`int`, optional, defaults to 16) — The size (resolution) of each patch.
- num_channels (`int`, optional, defaults to 3) — The number of input channels.
- qkv_bias (`bool`, optional, defaults to `True`) — Whether to add a bias to the queries, keys and values.
- encoder_stride (`int`, optional, defaults to 16) — Factor to increase the spatial resolution by in the decoder head for masked image modeling.
- pooler_output_size (`int`, optional) — Dimensionality of the pooler layer. If None, defaults to `hidden_size`.
- pooler_act (`str`, optional, defaults to `"tanh"`) — The activation function to be used by the pooler. Keys of ACT2FN are supported for Flax and Pytorch, and elements of https://www.tensorflow.org/api_docs/python/tf/keras/activations are supported for Tensorflow.

This is the configuration class to store the configuration of a DeiTModel. It is used to instantiate an DeiT model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the DeiT facebook/deit-base-distilled-patch16-224 architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import DeiTConfig, DeiTModel

    >>> # Initializing a DeiT deit-base-distilled-patch16-224 style configuration
    >>> configuration = DeiTConfig()

    >>> # Initializing a model (with random weights) from the deit-base-distilled-patch16-224 style configuration
    >>> model = DeiTModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
DeiTFeatureExtractor

###
class transformers.DeiTFeatureExtractor

< source >

( *args **kwargs )

####
__call__

< source >

( images **kwargs )

Preprocess an image or a batch of images.

##
DeiTImageProcessor

###
class transformers.DeiTImageProcessor

< source >

( do_resize: bool = True size: typing.Dict[str, int] = None resample: Resampling = 3 do_center_crop: bool = True crop_size: typing.Dict[str, int] = None rescale_factor: typing.Union[int, float] = 0.00392156862745098 do_rescale: bool = True do_normalize: bool = True image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None **kwargs )

Parameters

- do_resize (`bool`, optional, defaults to `True`) — Whether to resize the image’s (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in `preprocess`.
- size (`Dict[str, int]` optional, defaults to `{"height" -- 256, "width": 256}`): Size of the image after `resize`. Can be overridden by `size` in `preprocess`.
- resample (`PILImageResampling` filter, optional, defaults to `Resampling.BICUBIC`) — Resampling filter to use if resizing the image. Can be overridden by `resample` in `preprocess`.
- do_center_crop (`bool`, optional, defaults to `True`) — Whether to center crop the image. If the input size is smaller than `crop_size` along any edge, the image is padded with 0’s and then center cropped. Can be overridden by `do_center_crop` in `preprocess`.
- crop_size (`Dict[str, int]`, optional, defaults to `{"height" -- 224, "width": 224}`): Desired output size when applying center-cropping. Can be overridden by `crop_size` in `preprocess`.
- rescale_factor (`int` or `float`, optional, defaults to `1/255`) — Scale factor to use if rescaling the image. Can be overridden by the `rescale_factor` parameter in the `preprocess` method.
- do_rescale (`bool`, optional, defaults to `True`) — Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method.
- do_normalize (`bool`, optional, defaults to `True`) — Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
- image_mean (`float` or `List[float]`, optional, defaults to `IMAGENET_STANDARD_MEAN`) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method.
- image_std (`float` or `List[float]`, optional, defaults to `IMAGENET_STANDARD_STD`) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method.

Constructs a DeiT image processor.

####
preprocess

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] do_resize: typing.Optional[bool] = None size: typing.Dict[str, int] = None resample = None do_center_crop: typing.Optional[bool] = None crop_size: typing.Dict[str, int] = None do_rescale: typing.Optional[bool] = None rescale_factor: typing.Optional[float] = None do_normalize: typing.Optional[bool] = None image_mean: typing.Union[float, typing.List[float], NoneType] = None image_std: typing.Union[float, typing.List[float], NoneType] = None return_tensors: typing.Union[str, transformers.utils.generic.TensorType, NoneType] = None data_format: ChannelDimension = <ChannelDimension.FIRST: 'channels_first'> input_data_format: typing.Union[str, transformers.image_utils.ChannelDimension, NoneType] = None )

Parameters

- images (`ImageInput`) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`.
- do_resize (`bool`, optional, defaults to `self.do_resize`) — Whether to resize the image.
- size (`Dict[str, int]`, optional, defaults to `self.size`) — Size of the image after `resize`.
- resample (`PILImageResampling`, optional, defaults to `self.resample`) — PILImageResampling filter to use if resizing the image Only has an effect if `do_resize` is set to `True`.
- do_center_crop (`bool`, optional, defaults to `self.do_center_crop`) — Whether to center crop the image.
- crop_size (`Dict[str, int]`, optional, defaults to `self.crop_size`) — Size of the image after center crop. If one edge the image is smaller than `crop_size`, it will be padded with zeros and then cropped
- do_rescale (`bool`, optional, defaults to `self.do_rescale`) — Whether to rescale the image values between [0 - 1].
- rescale_factor (`float`, optional, defaults to `self.rescale_factor`) — Rescale factor to rescale the image by if `do_rescale` is set to `True`.
- do_normalize (`bool`, optional, defaults to `self.do_normalize`) — Whether to normalize the image.
- image_mean (`float` or `List[float]`, optional, defaults to `self.image_mean`) — Image mean.
- image_std (`float` or `List[float]`, optional, defaults to `self.image_std`) — Image standard deviation.
- return_tensors (`str` or `TensorType`, optional) — The type of tensors to return. Can be one of:
  - `None`: Return a list of `np.ndarray`.
  - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
  - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
  - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
  - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
- data_format (`ChannelDimension` or `str`, optional, defaults to `ChannelDimension.FIRST`) — The channel dimension format for the output image. Can be one of:
  - `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `ChannelDimension.LAST`: image in (height, width, num_channels) format.
- input_data_format (`ChannelDimension` or `str`, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

Preprocess an image or batch of images.

##
DeiTImageProcessorFast

###
class transformers.DeiTImageProcessorFast

< source >

( **kwargs: typing_extensions.Unpack[transformers.image_processing_utils_fast.DefaultFastImageProcessorKwargs] )

Parameters

- do_resize (`bool`, optional, defaults to `self.do_resize`) — Whether to resize the image’s (height, width) dimensions to the specified `size`. Can be overridden by the `do_resize` parameter in the `preprocess` method.
- size (`dict`, optional, defaults to `self.size`) — Size of the output image after resizing. Can be overridden by the `size` parameter in the `preprocess` method.
- default_to_square (`bool`, optional, defaults to `self.default_to_square`) — Whether to default to a square image when resizing, if size is an int.
- resample (`PILImageResampling`, optional, defaults to `self.resample`) — Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `resample` parameter in the `preprocess` method.
- do_center_crop (`bool`, optional, defaults to `self.do_center_crop`) — Whether to center crop the image to the specified `crop_size`. Can be overridden by `do_center_crop` in the `preprocess` method.
- crop_size (`Dict[str, int]` optional, defaults to `self.crop_size`) — Size of the output image after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` method.
- do_rescale (`bool`, optional, defaults to `self.do_rescale`) — Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method.
- rescale_factor (`int` or `float`, optional, defaults to `self.rescale_factor`) — Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be overridden by the `rescale_factor` parameter in the `preprocess` method.
- do_normalize (`bool`, optional, defaults to `self.do_normalize`) — Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
- image_mean (`float` or `List[float]`, optional, defaults to `self.image_mean`) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be overridden by the `image_mean` parameter in the `preprocess` method.
- image_std (`float` or `List[float]`, optional, defaults to `self.image_std`) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method.
- do_convert_rgb (`bool`, optional, defaults to `self.do_convert_rgb`) — Whether to convert the image to RGB.
- return_tensors (`str` or `TensorType`, optional, defaults to `self.return_tensors`) — Returns stacked tensors if set to `pt, otherwise returns a list of tensors.
- data_format (`ChannelDimension` or `str`, optional, defaults to `self.data_format`) — Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors.
- input_data_format (`ChannelDimension` or `str`, optional, defaults to `self.input_data_format`) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
- device (`torch.device`, optional, defaults to `self.device`) — The device to process the images on. If unset, the device is inferred from the input images.

Constructs a fast DeiT image processor.

####
preprocess

< source >

( images: typing.Union[ForwardRef('PIL.Image.Image'), numpy.ndarray, ForwardRef('torch.Tensor'), list['PIL.Image.Image'], list[numpy.ndarray], list['torch.Tensor']] **kwargs: typing_extensions.Unpack[transformers.image_processing_utils_fast.DefaultFastImageProcessorKwargs] )

Parameters

- images (`ImageInput`) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`.
- do_resize (`bool`, optional, defaults to `self.do_resize`) — Whether to resize the image.
- size (`Dict[str, int]`, optional, defaults to `self.size`) — Describes the maximum input dimensions to the model.
- resample (`PILImageResampling` or `InterpolationMode`, optional, defaults to `self.resample`) — Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`.
- do_center_crop (`bool`, optional, defaults to `self.do_center_crop`) — Whether to center crop the image.
- crop_size (`Dict[str, int]`, optional, defaults to `self.crop_size`) — Size of the output image after applying `center_crop`.
- do_rescale (`bool`, optional, defaults to `self.do_rescale`) — Whether to rescale the image.
- rescale_factor (`float`, optional, defaults to `self.rescale_factor`) — Rescale factor to rescale the image by if `do_rescale` is set to `True`.
- do_normalize (`bool`, optional, defaults to `self.do_normalize`) — Whether to normalize the image.
- image_mean (`float` or `List[float]`, optional, defaults to `self.image_mean`) — Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- image_std (`float` or `List[float]`, optional, defaults to `self.image_std`) — Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- do_convert_rgb (`bool`, optional, defaults to `self.do_convert_rgb`) — Whether to convert the image to RGB.
- return_tensors (`str` or `TensorType`, optional, defaults to `self.return_tensors`) — Returns stacked tensors if set to `pt, otherwise returns a list of tensors.
- data_format (`ChannelDimension` or `str`, optional, defaults to `self.data_format`) — Only `ChannelDimension.FIRST` is supported. Added for compatibility with slow processors.
- input_data_format (`ChannelDimension` or `str`, optional, defaults to `self.input_data_format`) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
- device (`torch.device`, optional, defaults to `self.device`) — The device to process the images on. If unset, the device is inferred from the input images.

Preprocess an image or batch of images.

Pytorch
Hide Pytorch content

##
DeiTModel

###
class transformers.DeiTModel

< source >

( config: DeiTConfig add_pooling_layer: bool = True use_mask_token: bool = False )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare DeiT Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: typing.Optional[torch.Tensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None interpolate_pos_encoding: bool = False ) →
transformers.modeling_outputs.BaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, optional) — Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).

Returns

transformers.modeling_outputs.BaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The DeiTModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, DeiTModel
    >>> import torch
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = DeiTModel.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(image, return_tensors="pt")

    >>> with torch.no_grad():
    ...     outputs = model(**inputs)

    >>> last_hidden_states = outputs.last_hidden_state
    >>> list(last_hidden_states.shape)
    [1, 198, 768]

##
DeiTForMaskedImageModeling

###
class transformers.DeiTForMaskedImageModeling

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model with a decoder on top for masked image modeling, as proposed in SimMIM.

Note that we provide a script to pre-train this model on custom data in our examples directory.

This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: typing.Optional[torch.Tensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None interpolate_pos_encoding: bool = False ) →
`transformers.modeling_outputs.MaskedImageModelingOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`) — Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).

Returns

`transformers.modeling_outputs.MaskedImageModelingOutput` or `tuple(torch.FloatTensor)`

A `transformers.modeling_outputs.MaskedImageModelingOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `bool_masked_pos` is provided) — Reconstruction loss.
- reconstruction (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Reconstructed / completed images.
- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or
- when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage.
- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when
- `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The DeiTForMaskedImageModeling forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling
    >>> import torch
    >>> from PIL import Image
    >>> import requests

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2
    >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values
    >>> # create random boolean mask of shape (batch_size, num_patches)
    >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool()

    >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
    >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction
    >>> list(reconstructed_pixel_values.shape)
    [1, 3, 224, 224]

##
DeiTForImageClassification

###
class transformers.DeiTForImageClassification

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet.

This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None interpolate_pos_encoding: bool = False ) →
transformers.modeling_outputs.ImageClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

transformers.modeling_outputs.ImageClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.ImageClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The DeiTForImageClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoImageProcessor, DeiTForImageClassification
    >>> import torch
    >>> from PIL import Image
    >>> import requests

    >>> torch.manual_seed(3)
    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here,
    >>> # so the head will be randomly initialized, hence the predictions will be random
    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(images=image, return_tensors="pt")
    >>> outputs = model(**inputs)
    >>> logits = outputs.logits
    >>> # model predicts one of the 1000 ImageNet classes
    >>> predicted_class_idx = logits.argmax(-1).item()
    >>> print("Predicted class:", model.config.id2label[predicted_class_idx])
    Predicted class: Polaroid camera, Polaroid Land camera

##
DeiTForImageClassificationWithTeacher

###
class transformers.DeiTForImageClassificationWithTeacher

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet.

.. warning::

This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported.

This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None interpolate_pos_encoding: bool = False ) →
`transformers.models.deit.modeling_deit.DeiTForImageClassificationWithTeacherOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.

Returns

`transformers.models.deit.modeling_deit.DeiTForImageClassificationWithTeacherOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.deit.modeling_deit.DeiTForImageClassificationWithTeacherOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Prediction scores as the average of the cls_logits and distillation logits.
- cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token).
- distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token).
- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs.
- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The DeiTForImageClassificationWithTeacher forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, DeiTForImageClassificationWithTeacher
    >>> import torch
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = DeiTForImageClassificationWithTeacher.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(image, return_tensors="pt")

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> # model predicts one of the 1000 ImageNet classes
    >>> predicted_label = logits.argmax(-1).item()
    >>> print(model.config.id2label[predicted_label])
    tabby, tabby cat

TensorFlow
Hide TensorFlow content

##
TFDeiTModel

###
class transformers.TFDeiTModel

< source >

( config: DeiTConfig add_pooling_layer: bool = True use_mask_token: bool = False **kwargs )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare DeiT Model transformer outputting raw hidden-states without any specific head on top. This model is a TensorFlow keras.layers.Layer. Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior.

####
call

< source >

( pixel_values: tf.Tensor | None = None bool_masked_pos: tf.Tensor | None = None head_mask: tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None interpolate_pos_encoding: bool = False training: bool = False ) →
transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or `tuple(tf.Tensor)`

Parameters

- pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFBaseModelOutputWithPooling or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`tf.Tensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFDeiTModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, TFDeiTModel
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = TFDeiTModel.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(image, return_tensors="tf")
    >>> outputs = model(**inputs)

    >>> last_hidden_states = outputs.last_hidden_state
    >>> list(last_hidden_states.shape)
    [1, 198, 768]

##
TFDeiTForMaskedImageModeling

###
class transformers.TFDeiTForMaskedImageModeling

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model with a decoder on top for masked image modeling, as proposed in SimMIM. This model is a TensorFlow keras.layers.Layer. Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior.

####
call

< source >

( pixel_values: tf.Tensor | None = None bool_masked_pos: tf.Tensor | None = None head_mask: tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None interpolate_pos_encoding: bool = False training: bool = False ) →
`transformers.modeling_tf_outputs.TFMaskedImageModelingOutput` or `tuple(tf.Tensor)`

Parameters

- pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- bool_masked_pos (`tf.Tensor` of type bool and shape `(batch_size, num_patches)`) — Boolean masked positions. Indicates which patches are masked (1) and which aren’t (0).

Returns

`transformers.modeling_tf_outputs.TFMaskedImageModelingOutput` or `tuple(tf.Tensor)`

A `transformers.modeling_tf_outputs.TFMaskedImageModelingOutput` or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- loss (`tf.Tensor` of shape `(1,)`, optional, returned when `bool_masked_pos` is provided) — Reconstruction loss.
- reconstruction (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`) — Reconstructed / completed images.
- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when
- `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage.
- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when
- `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFDeiTForMaskedImageModeling forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoImageProcessor, TFDeiTForMaskedImageModeling
    >>> import tensorflow as tf
    >>> from PIL import Image
    >>> import requests

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = TFDeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2
    >>> pixel_values = image_processor(images=image, return_tensors="tf").pixel_values
    >>> # create random boolean mask of shape (batch_size, num_patches)
    >>> bool_masked_pos = tf.cast(tf.random.uniform((1, num_patches), minval=0, maxval=2, dtype=tf.int32), tf.bool)

    >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
    >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction
    >>> list(reconstructed_pixel_values.shape)
    [1, 3, 224, 224]

##
TFDeiTForImageClassification

###
class transformers.TFDeiTForImageClassification

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet.

This model is a TensorFlow keras.layers.Layer. Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior.

####
call

< source >

( pixel_values: tf.Tensor | None = None head_mask: tf.Tensor | None = None labels: tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None interpolate_pos_encoding: bool = False training: bool = False ) →
`transformers.modeling_tf_outputs.TFImageClassifierOutput` or `tuple(tf.Tensor)`

Parameters

- pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- labels (`tf.Tensor` of shape `(batch_size,)`, optional) — Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

`transformers.modeling_tf_outputs.TFImageClassifierOutput` or `tuple(tf.Tensor)`

A `transformers.modeling_tf_outputs.TFImageClassifierOutput` or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- loss (`tf.Tensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFDeiTForImageClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoImageProcessor, TFDeiTForImageClassification
    >>> import tensorflow as tf
    >>> from PIL import Image
    >>> import requests

    >>> keras.utils.set_random_seed(3)
    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> # note: we are loading a TFDeiTForImageClassificationWithTeacher from the hub here,
    >>> # so the head will be randomly initialized, hence the predictions will be random
    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = TFDeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(images=image, return_tensors="tf")
    >>> outputs = model(**inputs)
    >>> logits = outputs.logits
    >>> # model predicts one of the 1000 ImageNet classes
    >>> predicted_class_idx = tf.math.argmax(logits, axis=-1)[0]
    >>> print("Predicted class:", model.config.id2label[int(predicted_class_idx)])
    Predicted class: little blue heron, Egretta caerulea

##
TFDeiTForImageClassificationWithTeacher

###
class transformers.TFDeiTForImageClassificationWithTeacher

< source >

( config: DeiTConfig )

Parameters

- config (DeiTConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet.

.. warning::

This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported.

This model is a TensorFlow keras.layers.Layer. Use it as a regular TensorFlow Module and refer to the TensorFlow documentation for all matter related to general usage and behavior.

####
call

< source >

( pixel_values: tf.Tensor | None = None head_mask: tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None interpolate_pos_encoding: bool = False training: bool = False ) →
`transformers.models.deit.modeling_tf_deit.TFDeiTForImageClassificationWithTeacherOutput` or `tuple(tf.Tensor)`

Parameters

- pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See DeiTImageProcessor.call() for details.
- head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional, defaults to `False`) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.deit.modeling_tf_deit.TFDeiTForImageClassificationWithTeacherOutput` or `tuple(tf.Tensor)`

A `transformers.models.deit.modeling_tf_deit.TFDeiTForImageClassificationWithTeacherOutput` or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (DeiTConfig) and inputs.

- logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`) — Prediction scores as the average of the cls_logits and distillation logits.
- cls_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`) — Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token).
- distillation_logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`) — Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token).
- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs.
- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFDeiTForImageClassificationWithTeacher forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, TFDeiTForImageClassificationWithTeacher
    >>> import tensorflow as tf
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224")
    >>> model = TFDeiTForImageClassificationWithTeacher.from_pretrained("facebook/deit-base-distilled-patch16-224")

    >>> inputs = image_processor(image, return_tensors="tf")
    >>> logits = model(**inputs).logits

    >>> # model predicts one of the 1000 ImageNet classes
    >>> predicted_label = int(tf.math.argmax(logits, axis=-1))
    >>> print(model.config.id2label[predicted_label])
    tabby, tabby cat

< > Update on GitHub