#
RoBERTa

##
개요

RoBERTa 모델은 Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov가 제안한 논문 RoBERTa: A Robustly Optimized BERT Pretraining Approach에서 소개되었습니다. 이 모델은 2018년에 구글에서 발표한 BERT 모델을 기반으로 합니다.

RoBERTa는 BERT를 기반으로 하며, 주요 하이퍼파라미터를 수정하고, 사전 학습 단계에서 다음 문장 예측(Next Sentence Prediction)을 제거했으며, 훨씬 더 큰 미니 배치 크기와 학습률을 사용하여 학습을 진행했습니다.

해당 논문의 초록입니다:

언어 모델 사전 학습은 성능을 크게 향상시켰지만, 서로 다른 접근 방식을 면밀히 비교하는 것은 어렵습니다. 학습은 계산 비용이 많이 들고, 종종 크기가 서로 다른 비공개 데이터셋에서 수행되며, 본 논문에서 보여주듯이 하이퍼파라미터 선택이 최종 성능에 큰 영향을 미칩니다. 우리는 BERT 사전 학습(Devlin et al., 2019)에 대한 재현 연구를 수행하여, 여러 핵심 하이퍼파라미터와 학습 데이터 크기의 영향을 면밀히 측정하였습니다. 그 결과, BERT는 충분히 학습되지 않았으며, 이후 발표된 모든 모델의 성능을 맞추거나 능가할 수 있음을 발견했습니다. 우리가 제안한 최상의 모델은 GLUE, RACE, SQuAD에서 최고 성능(state-of-the-art)을 달성했습니다. 이 결과는 지금까지 간과되어 온 설계 선택의 중요성을 강조하며, 최근 보고된 성능 향상의 근원이 무엇인지에 대한 의문을 제기합니다. 우리는 본 연구에서 사용한 모델과 코드를 공개합니다.

이 모델은 julien-c가 기여하였습니다. 원본 코드는 여기에서 확인할 수 있습니다.

##
사용 팁

- 이 구현은 BertModel과 동일하지만, 임베딩 부분에 약간의 수정이 있으며 RoBERTa 사전학습 모델에 맞게 설정되어 있습니다.

- RoBERTa는 BERT와 동일한 아키텍처를 가지고 있지만, 토크나이저로 바이트 수준 BPE(Byte-Pair Encoding, GPT-2와 동일)를 사용하고, 사전학습 방식이 다릅니다.

- RoBERTa는 `token_type_ids`를 사용하지 않기 때문에, 어떤 토큰이 어떤 문장(segment)에 속하는지 별도로 표시할 필요가 없습니다. 문장 구분은 분리 토큰 `tokenizer.sep_token`(또는 `</s>`)을 사용해 나누면 됩니다.

- RoBERTa는 BERT와 유사하지만, 더 나은 사전학습 기법을 사용합니다:

  - 동적 마스킹: RoBERTa는 매 에폭마다 토큰을 다르게 마스킹하는 반면, BERT는 한 번만 마스킹합니다.
  - 문장 패킹: 여러 문장을 최대 512 토큰까지 함께 패킹하여, 문장이 여러 문서에 걸쳐 있을 수도 있습니다.
  - 더 큰 배치 사이즈: 학습 시 더 큰 미니배치를 사용합니다.
  - 바이트 수준 BPE 어휘: 문자를 단위로 하지 않고 바이트 단위로 BPE를 적용하여 유니코드 문자를 더 유연하게 처리할 수 있습니다.

- CamemBERT은 RoBERTa를 기반으로 한 래퍼 모델입니다. 사용 예제는 해당 모델 페이지를 참고하세요.

##
자료

RoBERTa를 처음 다룰 때 도움이 되는 Hugging Face 공식 자료와 커뮤니티 자료(🌎 아이콘으로 표시됨) 목록입니다. 이 목록에 자료를 추가하고 싶다면 언제든지 Pull Request를 보내주세요! 저희가 검토 후 반영하겠습니다. 추가하려는 자료는 기존 자료를 단순히 복제하는 것이 아닌, 새롭거나 유의미한 내용을 포함하고 있는 것이 좋습니다.

Text Classification

- RoBERTa와 Inference API를 활용한 트위터 감성 분석 시작하기 블로그 포스트.
- RoBERTa를 활용한 Kili 및 Hugging Face AutoTrain을 이용한 의견 분류에 관한 블로그 포스트.
- 감성 분석을 위한 RoBERTa 미세조정을 하는 방법에 대한 노트북.🌎
- [‘RobertaForSequenceClassification’]은 예제 스크립트와 노트북에서 지원됩니다.
- TFRobertaForSequenceClassification는 예제 스크립트와 노트북에서 지원됩니다.
- FlaxRobertaForSequenceClassification는 예제 스크립트와 노트북에서 지원됩니다.
- 텍스트 분류 작업 가이드

Token Classification

- RobertaForTokenClassification은 예제 스크립트와 노트북에서 지원됩니다.
- TFRobertaForTokenClassification은 예제 스크립트와 노트북에서 지원됩니다.
- FlaxRobertaForTokenClassification는 예제 스크립트에서 지원됩니다.
- 🤗 Hugging Face 코스의 토큰 분류 챕터
- 토큰 분류 작업 가이드

Fill-Mask

- RoBERTa를 활용한 Transformers와 Tokenizers를 활용한 새로운 언어 모델을 처음부터 학습하는 방법에 대한 블로그 포스트.
- RobertaForMaskedLM은 예제 스크립트와 노트북에서 지원됩니다.
- TFRobertaForMaskedLM은 예제 스크립트와 노트북에서 지원됩니다.
- FlaxRobertaForMaskedLM은 예제 스크립트와 노트북에서 지원됩니다.
- 🤗 Hugging Face 코스의 마스킹 언어 모델링 챕터
- 마스킹 언어 모델링 작업 가이드

Question Answering

- RoBERTa를 활용한 질문 응답 작업에서의 Optimum과 Transformers 파이프라인을 이용한 추론 가속화에 대한 블로그 포스트.
- RobertaForQuestionAnswering은 예제 스크립트와 노트북에서 지원됩니다.
- TFRobertaForQuestionAnswering은 예제 스크립트와 노트북에서 지원됩니다.
- FlaxRobertaForQuestionAnswering은 예제 스크립트에서 지원됩니다.
- 🤗 Hugging Face 코스의 질의응답 챕터
- 질의응답 작업 가이드

다중 선택

- RobertaForMultipleChoice는 예제 스크립트와 노트북에서 지원됩니다.
- TFRobertaForMultipleChoice는 예제 스크립트와 노트북에서 지원됩니다.
- 다중 선택 작업 가이드

##
RobertaConfig

###
class transformers.RobertaConfig

< source >

( vocab_size = 50265 hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.1 attention_probs_dropout_prob = 0.1 max_position_embeddings = 512 type_vocab_size = 2 initializer_range = 0.02 layer_norm_eps = 1e-12 pad_token_id = 1 bos_token_id = 0 eos_token_id = 2 position_embedding_type = 'absolute' use_cache = True classifier_dropout = None **kwargs )

Parameters

- vocab_size (`int`, optional, defaults to 50265) — Vocabulary size of the RoBERTa model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling RobertaModel or TFRobertaModel.
- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- intermediate_size (`int`, optional, defaults to 3072) — Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
- hidden_act (`str` or `Callable`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"silu"` and `"gelu_new"` are supported.
- hidden_dropout_prob (`float`, optional, defaults to 0.1) — The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
- attention_probs_dropout_prob (`float`, optional, defaults to 0.1) — The dropout ratio for the attention probabilities.
- max_position_embeddings (`int`, optional, defaults to 512) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
- type_vocab_size (`int`, optional, defaults to 2) — The vocabulary size of the `token_type_ids` passed when calling RobertaModel or TFRobertaModel.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- layer_norm_eps (`float`, optional, defaults to 1e-12) — The epsilon used by the layer normalization layers.
- position_embedding_type (`str`, optional, defaults to `"absolute"`) — Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to Self-Attention with Relative Position Representations (Shaw et al.). For more information on `"relative_key_query"`, please refer to Method 4 in Improve Transformer Models with Better Relative Position Embeddings (Huang et al.).
- is_decoder (`bool`, optional, defaults to `False`) — Whether the model is used as a decoder or not. If `False`, the model is used as an encoder.
- use_cache (`bool`, optional, defaults to `True`) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`.
- classifier_dropout (`float`, optional) — The dropout ratio for the classification head.

This is the configuration class to store the configuration of a RobertaModel or a TFRobertaModel. It is used to instantiate a RoBERTa model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the RoBERTa FacebookAI/roberta-base architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Examples:

    >>> from transformers import RobertaConfig, RobertaModel

    >>> # Initializing a RoBERTa configuration
    >>> configuration = RobertaConfig()

    >>> # Initializing a model (with random weights) from the configuration
    >>> model = RobertaModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
RobertaTokenizer

###
class transformers.RobertaTokenizer

< source >

( vocab_file merges_file errors = 'replace' bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' add_prefix_space = False **kwargs )

Parameters

- vocab_file (`str`) — Path to the vocabulary file.
- merges_file (`str`) — Path to the merges file.
- errors (`str`, optional, defaults to `"replace"`) — Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information.
- bos_token (`str`, optional, defaults to `"<s>"`) — The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

  When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`.

- eos_token (`str`, optional, defaults to `"</s>"`) — The end of sequence token.

  When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`.

- sep_token (`str`, optional, defaults to `"</s>"`) — The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens.
- cls_token (`str`, optional, defaults to `"<s>"`) — The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
- unk_token (`str`, optional, defaults to `"<unk>"`) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
- pad_token (`str`, optional, defaults to `"<pad>"`) — The token used for padding, for example when batching sequences of different lengths.
- mask_token (`str`, optional, defaults to `"<mask>"`) — The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
- add_prefix_space (`bool`, optional, defaults to `False`) — Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (RoBERTa tokenizer detect beginning of words by the preceding space).

Constructs a RoBERTa tokenizer, derived from the GPT-2 tokenizer, using byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will

be encoded differently whether it is at the beginning of the sentence (without space) or not:

    >>> from transformers import RobertaTokenizer

    >>> tokenizer = RobertaTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> tokenizer("Hello world")["input_ids"]
    [0, 31414, 232, 2]

    >>> tokenizer(" Hello world")["input_ids"]
    [0, 20920, 232, 2]

You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.

When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one).

This tokenizer inherits from `PreTrainedTokenizer` which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

####
build_inputs_with_special_tokens

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs to which the special tokens will be added.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of input IDs with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A RoBERTa sequence has the following format:

- single sequence: `<s> X </s>`
- pair of sequences: `<s> A </s></s> B </s>`

####
get_special_tokens_mask

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None already_has_special_tokens: bool = False ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.
- already_has_special_tokens (`bool`, optional, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

Returns

`List[int]`

A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

####
create_token_type_ids_from_sequences

< source >

( token_ids_0: typing.List[int] token_ids_1: typing.Optional[typing.List[int]] = None ) →
`List[int]`

Parameters

- token_ids_0 (`List[int]`) — List of IDs.
- token_ids_1 (`List[int]`, optional) — Optional second list of IDs for sequence pairs.

Returns

`List[int]`

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. RoBERTa does not make use of token type ids, therefore a list of zeros is returned.

####
save_vocabulary

< source >

( save_directory: str filename_prefix: typing.Optional[str] = None )

##
RobertaTokenizerFast

###
class transformers.RobertaTokenizerFast

< source >

( vocab_file = None merges_file = None tokenizer_file = None errors = 'replace' bos_token = '<s>' eos_token = '</s>' sep_token = '</s>' cls_token = '<s>' unk_token = '<unk>' pad_token = '<pad>' mask_token = '<mask>' add_prefix_space = False trim_offsets = True **kwargs )

Parameters

- vocab_file (`str`) — Path to the vocabulary file.
- merges_file (`str`) — Path to the merges file.
- errors (`str`, optional, defaults to `"replace"`) — Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information.
- bos_token (`str`, optional, defaults to `"<s>"`) — The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

  When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`.

- eos_token (`str`, optional, defaults to `"</s>"`) — The end of sequence token.

  When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`.

- sep_token (`str`, optional, defaults to `"</s>"`) — The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens.
- cls_token (`str`, optional, defaults to `"<s>"`) — The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
- unk_token (`str`, optional, defaults to `"<unk>"`) — The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.
- pad_token (`str`, optional, defaults to `"<pad>"`) — The token used for padding, for example when batching sequences of different lengths.
- mask_token (`str`, optional, defaults to `"<mask>"`) — The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict.
- add_prefix_space (`bool`, optional, defaults to `False`) — Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
- trim_offsets (`bool`, optional, defaults to `True`) — Whether the post processing step should trim offsets to avoid including whitespaces.

Construct a “fast” RoBERTa tokenizer (backed by HuggingFace’s tokenizers library), derived from the GPT-2 tokenizer, using byte-level Byte-Pair-Encoding.

This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will

be encoded differently whether it is at the beginning of the sentence (without space) or not:

    >>> from transformers import RobertaTokenizerFast

    >>> tokenizer = RobertaTokenizerFast.from_pretrained("FacebookAI/roberta-base")
    >>> tokenizer("Hello world")["input_ids"]
    [0, 31414, 232, 2]

    >>> tokenizer(" Hello world")["input_ids"]
    [0, 20920, 232, 2]

You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.

When used with `is_split_into_words=True`, this tokenizer needs to be instantiated with `add_prefix_space=True`.

This tokenizer inherits from `PreTrainedTokenizerFast` which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

####
build_inputs_with_special_tokens

< source >

( token_ids_0 token_ids_1 = None )

Pytorch
Hide Pytorch content

##
RobertaModel

###
class transformers.RobertaModel

< source >

( config add_pooling_layer = True )

Parameters

- config (RobertaModel) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
- add_pooling_layer (`bool`, optional, defaults to `True`) — Whether to add a pooling layer

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None token_type_ids: typing.Optional[torch.Tensor] = None position_ids: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None inputs_embeds: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None encoder_attention_mask: typing.Optional[torch.Tensor] = None past_key_values: typing.Optional[typing.List[torch.FloatTensor]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- encoder_hidden_states (`torch.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- past_key_values (`List[torch.FloatTensor]`, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.

  Two formats are allowed:

  - a Cache instance, see our kv cache guide;
  - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format.

  The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned.

  If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don’t have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`.

- use_cache (`bool`, optional) — If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`).
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- cross_attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` and `config.add_cross_attention=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.

- past_key_values (`tuple(tuple(torch.FloatTensor))`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.

  Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

The RobertaModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

##
RobertaForCausalLM

###
class transformers.RobertaForCausalLM

< source >

( config )

Parameters

- config (RobertaForCausalLM) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model with a `language modeling` head on top for CLM fine-tuning.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None use_cache: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None **kwargs ) →
transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
- past_key_values (`Tuple[Tuple[torch.FloatTensor]]`, optional) — Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`.

  Two formats are allowed:

  - a Cache instance, see our kv cache guide;
  - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format.

  The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned.

  If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don’t have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`.

- use_cache (`bool`, optional) — If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`).
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Language modeling loss (for next-token prediction).

- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- cross_attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Cross attentions weights after the attention softmax, used to compute the weighted average in the cross-attention heads.

- past_key_values (`tuple(tuple(torch.FloatTensor))`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — Tuple of `torch.FloatTensor` tuples of length `config.n_layers`, with each tuple containing the cached key, value states of the self-attention and the cross-attention layers if model is used in encoder-decoder setting. Only relevant if `config.is_decoder = True`.

  Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

The RobertaForCausalLM forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base")
    >>> config.is_decoder = True
    >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config)

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
    >>> outputs = model(**inputs)

    >>> prediction_logits = outputs.logits

##
RobertaForMaskedLM

###
class transformers.RobertaForMaskedLM

< source >

( config )

Parameters

- config (RobertaForMaskedLM) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Roberta Model with a `language modeling` head on top.”

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None encoder_hidden_states: typing.Optional[torch.FloatTensor] = None encoder_attention_mask: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.MaskedLMOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.MaskedLMOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.MaskedLMOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Masked language modeling (MLM) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The RobertaForMaskedLM forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, RobertaForMaskedLM
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForMaskedLM.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("The capital of France is <mask>.", return_tensors="pt")

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> # retrieve index of <mask>
    >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

    >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
    >>> tokenizer.decode(predicted_token_id)
    ...

    >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]
    >>> # mask labels of non-<mask> tokens
    >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)

    >>> outputs = model(**inputs, labels=labels)
    >>> round(outputs.loss.item(), 2)
    ...

##
RobertaForSequenceClassification

###
class transformers.RobertaForSequenceClassification

< source >

( config )

Parameters

- config (RobertaForSequenceClassification) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.SequenceClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.SequenceClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.SequenceClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The RobertaForSequenceClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example of single-label classification:

    >>> import torch
    >>> from transformers import AutoTokenizer, RobertaForSequenceClassification

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForSequenceClassification.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> predicted_class_id = logits.argmax().item()
    >>> model.config.id2label[predicted_class_id]
    ...

    >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
    >>> num_labels = len(model.config.id2label)
    >>> model = RobertaForSequenceClassification.from_pretrained("FacebookAI/roberta-base", num_labels=num_labels)

    >>> labels = torch.tensor([1])
    >>> loss = model(**inputs, labels=labels).loss
    >>> round(loss.item(), 2)
    ...

Example of multi-label classification:

    >>> import torch
    >>> from transformers import AutoTokenizer, RobertaForSequenceClassification

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForSequenceClassification.from_pretrained("FacebookAI/roberta-base", problem_type="multi_label_classification")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]

    >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
    >>> num_labels = len(model.config.id2label)
    >>> model = RobertaForSequenceClassification.from_pretrained(
    ...     "FacebookAI/roberta-base", num_labels=num_labels, problem_type="multi_label_classification"
    ... )

    >>> labels = torch.sum(
    ...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
    ... ).to(torch.float)
    >>> loss = model(**inputs, labels=labels).loss

##
RobertaForMultipleChoice

###
class transformers.RobertaForMultipleChoice

< source >

( config )

Parameters

- config (RobertaForMultipleChoice) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Roberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.MultipleChoiceModelOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above)
- position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.MultipleChoiceModelOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.MultipleChoiceModelOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape (1,), optional, returned when `labels` is provided) — Classification loss.

- logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`) — num_choices is the second dimension of the input tensors. (see input_ids above).

  Classification scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The RobertaForMultipleChoice forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, RobertaForMultipleChoice
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForMultipleChoice.from_pretrained("FacebookAI/roberta-base")

    >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
    >>> choice0 = "It is eaten with a fork and a knife."
    >>> choice1 = "It is eaten while held in the hand."
    >>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1

    >>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
    >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1

    >>> # the linear classifier still needs to be trained
    >>> loss = outputs.loss
    >>> logits = outputs.logits

##
RobertaForTokenClassification

###
class transformers.RobertaForTokenClassification

< source >

( config )

Parameters

- config (RobertaForTokenClassification) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Roberta transformer with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None labels: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.TokenClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.TokenClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.TokenClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Classification loss.

- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`) — Classification scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The RobertaForTokenClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, RobertaForTokenClassification
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForTokenClassification.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer(
    ...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
    ... )

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> predicted_token_class_ids = logits.argmax(-1)

    >>> # Note that tokens are classified rather then input words which means that
    >>> # there might be more predicted token classes than words.
    >>> # Multiple token classes might account for the same word
    >>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
    >>> predicted_tokens_classes
    ...

    >>> labels = predicted_token_class_ids
    >>> loss = model(**inputs, labels=labels).loss
    >>> round(loss.item(), 2)
    ...

##
RobertaForQuestionAnswering

###
class transformers.RobertaForQuestionAnswering

< source >

( config )

Parameters

- config (RobertaForQuestionAnswering) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Roberta transformer with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layer on top of the hidden-states output to compute `span start logits` and `span end logits`).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[torch.FloatTensor] = None token_type_ids: typing.Optional[torch.LongTensor] = None position_ids: typing.Optional[torch.LongTensor] = None head_mask: typing.Optional[torch.FloatTensor] = None inputs_embeds: typing.Optional[torch.FloatTensor] = None start_positions: typing.Optional[torch.LongTensor] = None end_positions: typing.Optional[torch.LongTensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) →
transformers.modeling_outputs.QuestionAnsweringModelOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value

    > = 2. All the value in this tensor should be always < type_vocab_size.

  What are token type IDs?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`.

  What are position IDs?

- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- start_positions (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss.
- end_positions (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_outputs.QuestionAnsweringModelOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `labels` is provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.

- start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`) — Span-start scores (before SoftMax).

- end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`) — Span-end scores (before SoftMax).

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The RobertaForQuestionAnswering forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, RobertaForQuestionAnswering
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = RobertaForQuestionAnswering.from_pretrained("FacebookAI/roberta-base")

    >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

    >>> inputs = tokenizer(question, text, return_tensors="pt")
    >>> with torch.no_grad():
    ...     outputs = model(**inputs)

    >>> answer_start_index = outputs.start_logits.argmax()
    >>> answer_end_index = outputs.end_logits.argmax()

    >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
    >>> tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)
    ...

    >>> # target is "nice puppet"
    >>> target_start_index = torch.tensor([14])
    >>> target_end_index = torch.tensor([15])

    >>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
    >>> loss = outputs.loss
    >>> round(loss.item(), 2)
    ...

TensorFlow
Hide TensorFlow content

##
TFRobertaModel

###
class transformers.TFRobertaModel

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare RoBERTa Model transformer outputting raw hidden-states without any specific head on top.

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None encoder_hidden_states: np.ndarray | tf.Tensor | None = None encoder_attention_mask: np.ndarray | tf.Tensor | None = None past_key_values: Optional[Tuple[Tuple[Union[np.ndarray, tf.Tensor]]]] = None use_cache: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- encoder_hidden_states (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- past_key_values (`Tuple[Tuple[tf.Tensor]]` of length `config.n_layers`) — contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don’t have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`.
- use_cache (`bool`, optional, defaults to `True`) — If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Set to `False` during training, `True` during generation

Returns

transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`tf.Tensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

  This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.

- past_key_values (`List[tf.Tensor]`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — List of `tf.Tensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`).

  Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- cross_attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.

The TFRobertaModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaModel
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = TFRobertaModel.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
    >>> outputs = model(inputs)

    >>> last_hidden_states = outputs.last_hidden_state

##
TFRobertaForCausalLM

###
class transformers.TFRobertaForCausalLM

< source >

( config: RobertaConfig *inputs **kwargs )

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None encoder_hidden_states: np.ndarray | tf.Tensor | None = None encoder_attention_mask: np.ndarray | tf.Tensor | None = None past_key_values: Optional[Tuple[Tuple[Union[np.ndarray, tf.Tensor]]]] = None use_cache: Optional[bool] = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFCausalLMOutputWithCrossAttentions or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- encoder_hidden_states (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder.
- encoder_attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.
- past_key_values (`Tuple[Tuple[tf.Tensor]]` of length `config.n_layers`) — contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don’t have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`.
- use_cache (`bool`, optional, defaults to `True`) — If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Set to `False` during training, `True` during generation
- labels (`tf.Tensor` or `np.ndarray` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the cross entropy classification loss. Indices should be in `[0, ..., config.vocab_size - 1]`.

Returns

transformers.modeling_tf_outputs.TFCausalLMOutputWithCrossAttentions or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFCausalLMOutputWithCrossAttentions or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape `(n,)`, optional, where n is the number of non-masked labels, returned when `labels` is provided) — Language modeling loss (for next-token prediction).

- logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- cross_attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.

- past_key_values (`List[tf.Tensor]`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — List of `tf.Tensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`).

  Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

The TFRobertaForCausalLM forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForCausalLM
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = TFRobertaForCausalLM.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")
    >>> outputs = model(inputs)
    >>> logits = outputs.logits

##
TFRobertaForMaskedLM

###
class transformers.TFRobertaForMaskedLM

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model with a `language modeling` head on top.

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFMaskedLMOutput or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`

Returns

transformers.modeling_tf_outputs.TFMaskedLMOutput or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFMaskedLMOutput or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape `(n,)`, optional, where n is the number of non-masked labels, returned when `labels` is provided) — Masked language modeling (MLM) loss.

- logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFRobertaForMaskedLM forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForMaskedLM
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = TFRobertaForMaskedLM.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("The capital of France is <mask>.", return_tensors="tf")
    >>> logits = model(**inputs).logits

    >>> # retrieve index of <mask>
    >>> mask_token_index = tf.where((inputs.input_ids == tokenizer.mask_token_id)[0])
    >>> selected_logits = tf.gather_nd(logits[0], indices=mask_token_index)

    >>> predicted_token_id = tf.math.argmax(selected_logits, axis=-1)
    >>> tokenizer.decode(predicted_token_id)
    ' Paris'

    >>> labels = tokenizer("The capital of France is Paris.", return_tensors="tf")["input_ids"]
    >>> # mask labels of non-<mask> tokens
    >>> labels = tf.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)

    >>> outputs = model(**inputs, labels=labels)
    >>> round(float(outputs.loss), 2)
    0.1

##
TFRobertaForSequenceClassification

###
class transformers.TFRobertaForSequenceClassification

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFSequenceClassifierOutput or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- labels (`tf.Tensor` of shape `(batch_size,)`, optional) — Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

transformers.modeling_tf_outputs.TFSequenceClassifierOutput or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFSequenceClassifierOutput or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape `(batch_size, )`, optional, returned when `labels` is provided) — Classification (or regression if config.num_labels==1) loss.

- logits (`tf.Tensor` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFRobertaForSequenceClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForSequenceClassification
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("cardiffnlp/twitter-roberta-base-emotion")
    >>> model = TFRobertaForSequenceClassification.from_pretrained("cardiffnlp/twitter-roberta-base-emotion")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")

    >>> logits = model(**inputs).logits

    >>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
    >>> model.config.id2label[predicted_class_id]
    'optimism'

    >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
    >>> num_labels = len(model.config.id2label)
    >>> model = TFRobertaForSequenceClassification.from_pretrained("cardiffnlp/twitter-roberta-base-emotion", num_labels=num_labels)

    >>> labels = tf.constant(1)
    >>> loss = model(**inputs, labels=labels).loss
    >>> round(float(loss), 2)
    0.08

##
TFRobertaForMultipleChoice

###
class transformers.TFRobertaForMultipleChoice

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, num_choices, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, num_choices, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, num_choices, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, num_choices, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- labels (`tf.Tensor` of shape `(batch_size,)`, optional) — Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above)

Returns

transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape (batch_size, ), optional, returned when `labels` is provided) — Classification loss.

- logits (`tf.Tensor` of shape `(batch_size, num_choices)`) — num_choices is the second dimension of the input tensors. (see input_ids above).

  Classification scores (before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFRobertaForMultipleChoice forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForMultipleChoice
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = TFRobertaForMultipleChoice.from_pretrained("FacebookAI/roberta-base")

    >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
    >>> choice0 = "It is eaten with a fork and a knife."
    >>> choice1 = "It is eaten while held in the hand."

    >>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="tf", padding=True)
    >>> inputs = {k: tf.expand_dims(v, 0) for k, v in encoding.items()}
    >>> outputs = model(inputs)  # batch size is 1

    >>> # the linear classifier still needs to be trained
    >>> logits = outputs.logits

##
TFRobertaForTokenClassification

###
class transformers.TFRobertaForTokenClassification

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None labels: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFTokenClassifierOutput or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.

Returns

transformers.modeling_tf_outputs.TFTokenClassifierOutput or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFTokenClassifierOutput or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape `(n,)`, optional, where n is the number of unmasked labels, returned when `labels` is provided) — Classification loss.

- logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.num_labels)`) — Classification scores (before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFRobertaForTokenClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForTokenClassification
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("ydshieh/roberta-large-ner-english")
    >>> model = TFRobertaForTokenClassification.from_pretrained("ydshieh/roberta-large-ner-english")

    >>> inputs = tokenizer(
    ...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="tf"
    ... )

    >>> logits = model(**inputs).logits
    >>> predicted_token_class_ids = tf.math.argmax(logits, axis=-1)

    >>> # Note that tokens are classified rather then input words which means that
    >>> # there might be more predicted token classes than words.
    >>> # Multiple token classes might account for the same word
    >>> predicted_tokens_classes = [model.config.id2label[t] for t in predicted_token_class_ids[0].numpy().tolist()]
    >>> predicted_tokens_classes
    ['O', 'ORG', 'ORG', 'O', 'O', 'O', 'O', 'O', 'LOC', 'O', 'LOC', 'LOC']

    >>> labels = predicted_token_class_ids
    >>> loss = tf.math.reduce_mean(model(**inputs, labels=labels).loss)
    >>> round(float(loss), 2)
    0.01

##
TFRobertaForQuestionAnswering

###
class transformers.TFRobertaForQuestionAnswering

< source >

( config *inputs **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).

This model inherits from TFPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a keras.Model subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior.

TensorFlow models and layers in `transformers` accept two formats as input:

- having all inputs as keyword arguments (like PyTorch models), or
- having all inputs as a list, tuple or dict in the first positional argument.

The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should “just work” for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument:

- a single Tensor with `input_ids` only and nothing else: `model(input_ids)`
- a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])`
- a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})`

Note that when creating models and layers with subclassing then you don’t need to worry about any of this, as you can just pass inputs like you would to any other Python function!

####
call

< source >

( input_ids: TFModelInputType | None = None attention_mask: np.ndarray | tf.Tensor | None = None token_type_ids: np.ndarray | tf.Tensor | None = None position_ids: np.ndarray | tf.Tensor | None = None head_mask: np.ndarray | tf.Tensor | None = None inputs_embeds: np.ndarray | tf.Tensor | None = None output_attentions: Optional[bool] = None output_hidden_states: Optional[bool] = None return_dict: Optional[bool] = None start_positions: np.ndarray | tf.Tensor | None = None end_positions: np.ndarray | tf.Tensor | None = None training: Optional[bool] = False ) →
transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or `tuple(tf.Tensor)`

Parameters

- input_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.call() and PreTrainedTokenizer.encode() for details.

  What are input IDs?

- attention_mask (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`Numpy array` or `tf.Tensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, optional) — Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model’s internal embedding lookup matrix.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True.
- training (`bool`, optional, defaults to `False`) — Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation).
- start_positions (`tf.Tensor` of shape `(batch_size,)`, optional) — Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss.
- end_positions (`tf.Tensor` of shape `(batch_size,)`, optional) — Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss.

Returns

transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or `tuple(tf.Tensor)`

A transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or a tuple of `tf.Tensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- loss (`tf.Tensor` of shape `(batch_size, )`, optional, returned when `start_positions` and `end_positions` are provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.

- start_logits (`tf.Tensor` of shape `(batch_size, sequence_length)`) — Span-start scores (before SoftMax).

- end_logits (`tf.Tensor` of shape `(batch_size, sequence_length)`) — Span-end scores (before SoftMax).

- hidden_states (`tuple(tf.Tensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(tf.Tensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The TFRobertaForQuestionAnswering forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, TFRobertaForQuestionAnswering
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("ydshieh/roberta-base-squad2")
    >>> model = TFRobertaForQuestionAnswering.from_pretrained("ydshieh/roberta-base-squad2")

    >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

    >>> inputs = tokenizer(question, text, return_tensors="tf")
    >>> outputs = model(**inputs)

    >>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
    >>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])

    >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
    >>> tokenizer.decode(predict_answer_tokens)
    ' puppet'

    >>> # target is "nice puppet"
    >>> target_start_index = tf.constant([14])
    >>> target_end_index = tf.constant([15])

    >>> outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)
    >>> loss = tf.math.reduce_mean(outputs.loss)
    >>> round(float(loss), 2)
    0.86

JAX
Hide JAX content

##
FlaxRobertaModel

###
class transformers.FlaxRobertaModel

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The bare RoBERTa Model transformer outputting raw hidden-states without any specific head on top.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- last_hidden_state (`jnp.ndarray` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`jnp.ndarray` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaModel

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaModel.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")
    >>> outputs = model(**inputs)

    >>> last_hidden_states = outputs.last_hidden_state

##
FlaxRobertaForCausalLM

###
class transformers.FlaxRobertaForCausalLM

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model with a language modeling head on top (a linear layer on top of the hidden-states output) e.g for autoregressive tasks.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxCausalLMOutputWithCrossAttentions or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- logits (`jnp.ndarray` of shape `(batch_size, sequence_length, config.vocab_size)`) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- cross_attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Cross attentions weights after the attention softmax, used to compute the weighted average in the cross-attention heads.

- past_key_values (`tuple(tuple(jnp.ndarray))`, optional, returned when `use_cache=True` is passed or when `config.use_cache=True`) — Tuple of `jnp.ndarray` tuples of length `config.n_layers`, with each tuple containing the cached key, value states of the self-attention and the cross-attention layers if model is used in encoder-decoder setting. Only relevant if `config.is_decoder = True`.

  Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForCausalLM

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForCausalLM.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="np")
    >>> outputs = model(**inputs)

    >>> # retrieve logts for next token
    >>> next_token_logits = outputs.logits[:, -1]

##
FlaxRobertaForMaskedLM

###
class transformers.FlaxRobertaForMaskedLM

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

RoBERTa Model with a `language modeling` head on top.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxBaseModelOutputWithPooling or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- last_hidden_state (`jnp.ndarray` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`jnp.ndarray` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForMaskedLM

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForMaskedLM.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("The capital of France is [MASK].", return_tensors="jax")

    >>> outputs = model(**inputs)
    >>> logits = outputs.logits

##
FlaxRobertaForSequenceClassification

###
class transformers.FlaxRobertaForSequenceClassification

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxSequenceClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxSequenceClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxSequenceClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- logits (`jnp.ndarray` of shape `(batch_size, config.num_labels)`) — Classification (or regression if config.num_labels==1) scores (before SoftMax).

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForSequenceClassification

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForSequenceClassification.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")

    >>> outputs = model(**inputs)
    >>> logits = outputs.logits

##
FlaxRobertaForMultipleChoice

###
class transformers.FlaxRobertaForMultipleChoice

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, num_choices, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, num_choices, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, num_choices, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, num_choices, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, num_choices, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxMultipleChoiceModelOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- logits (`jnp.ndarray` of shape `(batch_size, num_choices)`) — num_choices is the second dimension of the input tensors. (see input_ids above).

  Classification scores (before SoftMax).

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForMultipleChoice

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForMultipleChoice.from_pretrained("FacebookAI/roberta-base")

    >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
    >>> choice0 = "It is eaten with a fork and a knife."
    >>> choice1 = "It is eaten while held in the hand."

    >>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="jax", padding=True)
    >>> outputs = model(**{k: v[None, :] for k, v in encoding.items()})

    >>> logits = outputs.logits

##
FlaxRobertaForTokenClassification

###
class transformers.FlaxRobertaForTokenClassification

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- logits (`jnp.ndarray` of shape `(batch_size, sequence_length, config.num_labels)`) — Classification scores (before SoftMax).

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForTokenClassification

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForTokenClassification.from_pretrained("FacebookAI/roberta-base")

    >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="jax")

    >>> outputs = model(**inputs)
    >>> logits = outputs.logits

##
FlaxRobertaForQuestionAnswering

###
class transformers.FlaxRobertaForQuestionAnswering

< source >

( config: RobertaConfig input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True gradient_checkpointing: bool = False **kwargs )

Parameters

- config (RobertaConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Roberta Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).

This model inherits from FlaxPreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading, saving and converting weights from PyTorch models)

This model is also a flax.linen.Module subclass. Use it as a regular Flax linen Module and refer to the Flax documentation for all matter related to general usage and behavior.

Finally, this model supports inherent JAX features such as:

- Just-In-Time (JIT) compilation
- Automatic Differentiation
- Vectorization
- Parallelization

####
__call__

< source >

( input_ids attention_mask = None token_type_ids = None position_ids = None head_mask = None encoder_hidden_states = None encoder_attention_mask = None params: typing.Optional[dict] = None dropout_rng: <function PRNGKey at 0x7f8f44315bd0> = None train: bool = False output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None past_key_values: typing.Optional[dict] = None ) →
transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- token_type_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`:
  - 0 corresponds to a sentence A token,
  - 1 corresponds to a sentence B token.

  What are token type IDs?

- position_ids (`numpy.ndarray` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.
- head_mask (`numpy.ndarray` of shape `(batch_size, sequence_length)`, `optional) -- Mask to nullify selected heads of the attention modules. Mask values selected in` [0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or `tuple(torch.FloatTensor)`

A transformers.modeling_flax_outputs.FlaxQuestionAnsweringModelOutput or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (RobertaConfig) and inputs.

- start_logits (`jnp.ndarray` of shape `(batch_size, sequence_length)`) — Span-start scores (before SoftMax).

- end_logits (`jnp.ndarray` of shape `(batch_size, sequence_length)`) — Span-end scores (before SoftMax).

- hidden_states (`tuple(jnp.ndarray)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `jnp.ndarray` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(jnp.ndarray)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `jnp.ndarray` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The `FlaxRobertaPreTrainedModel` forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoTokenizer, FlaxRobertaForQuestionAnswering

    >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base")
    >>> model = FlaxRobertaForQuestionAnswering.from_pretrained("FacebookAI/roberta-base")

    >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
    >>> inputs = tokenizer(question, text, return_tensors="jax")

    >>> outputs = model(**inputs)
    >>> start_scores = outputs.start_logits
    >>> end_scores = outputs.end_logits

< > Update on GitHub