clue/roberta_chinese_3L312_clue_tiny

Introduction

This model was trained on TPU and the details are as follows:

Model

Model_name	params	size	Training_corpus	Vocab
RoBERTa-tiny-clue Super_small_model	7.5M	28.3M	CLUECorpus2020	CLUEVocab
RoBERTa-tiny-pair Super_small_sentence_pair_model	7.5M	28.3M	CLUECorpus2020	CLUEVocab
RoBERTa-tiny3L768-clue small_model	38M	110M	CLUECorpus2020	CLUEVocab
RoBERTa-tiny3L312-clue small_model	<7.5M	24M	CLUECorpus2020	CLUEVocab
RoBERTa-large-clue Large_model	290M	1.20G	CLUECorpus2020	CLUEVocab
RoBERTa-large-pair Large_sentence_pair_model	290M	1.20G	CLUECorpus2020	CLUEVocab

Usage

With the help ofHuggingface-Transformers 2.5.1, you could use these model as follows

tokenizer = BertTokenizer.from_pretrained("MODEL_NAME")
model = BertModel.from_pretrained("MODEL_NAME")

MODEL_NAME：

Model_NAME	MODEL_LINK
RoBERTa-tiny-clue	clue/roberta_chinese_clue_tiny
RoBERTa-tiny-pair	clue/roberta_chinese_pair_tiny
RoBERTa-tiny3L768-clue	clue/roberta_chinese_3L768_clue_tiny
RoBERTa-tiny3L312-clue	clue/roberta_chinese_3L312_clue_tiny
RoBERTa-large-clue	clue/roberta_chinese_clue_large
RoBERTa-large-pair	clue/roberta_chinese_pair_large

Details

Please read https://arxiv.org/pdf/2003.01355.

Please visit our repository: https://github.com/CLUEbenchmark/CLUEPretrainedModels.git