Hugging Face's logo

ai-forever
/
sage-fredt5-large

Transformers
Safetensors
PyTorch
Russian
t5
text2text-generation
spellchecking
natural language generation
Eval Results
text-generation-inference
Model card Files
xet
Community
ai-forever commited on
Commit
0186562
·
verified ·
1 Parent(s): 9148b1f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +122 -0
README.md CHANGED
@@ -1,3 +1,125 @@
1
  ---
 
 
 
 
 
 
2
  license: mit
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - ru
4
+ tags:
5
+ - spellchecking
6
+ - pytorch
7
+ - natural language generation
8
  license: mit
9
+ datasets:
10
+ - ai-forever/spellcheck_punctuation_benchmark
11
+ metrics:
12
+ - precision
13
+ - recall
14
+ - f1
15
  ---
16
+ # sage-fredt5-large
17
+
18
+ ### Summary
19
+
20
+ The model corrects spelling errors and typos by bringing all the words in the text to the norm of the Russian language.
21
+ Corrector was trained based on the model [M2M100-1.2B](https://huggingface.co/facebook/m2m100_1.2B).
22
+ An extensive dataset with “artificial” errors was taken as a training corpus: the corpus was assembled on the basis of the Russian-language Wikipedia and transcripts of Russian-language videos, then typos and spelling errors were automatically introduced into it using the library [SAGE](https://github.com/ai-forever/sage).
23
+
24
+ ### Public references
25
+ - [SAGE library announcement](https://youtu.be/yFfkV0Qjuu0), DataFest 2023
26
+ - [Paper about synthetic error generation methods](https://www.dialog-21.ru/media/5914/martynovnplusetal056.pdf), Dialogue 2023
27
+ - [Paper about SAGE and our best solution](https://arxiv.org/abs/2308.09435), Review EACL 2024
28
+
29
+
30
+ ### Examples
31
+ | Input | Output |
32
+ | --- | --- |
33
+ | Думю ешцъа лет череа 10 ретроспективно просматривотьэ то будкетцц мне невероя тна ин те р но | Думаю что лет через 10 ретроспективно просматривать это будет мне невероятно интересно |
34
+ | Основая цель мероприятия - практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП, а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных проишествий, сокращение временных показателей реагирования. | Основная цель мероприятия - практическая отработка навыков по оказанию помощи гражданам, попавшим в ДТП, а также повышение и совершенствование уровня профессиональной подготовки сотрудников МЧС при проведении аварийно-спасательных работ по ликвидации последствий дорожно-транспортных происшествий, сокращение временных показателей реагирования. |
35
+ | прийдя в МГТУ я был удивлен никого необноружив там… | прийдя в МГТУ я был удивлен никого не обнаружив там... |
36
+ | | |
37
+
38
+ ## Metrics
39
+ ### Quality
40
+ Below are automatic metrics for determining the correctness of the spell checkers.
41
+ We compare our solution with both open automatic spell checkers and the ChatGPT family of models on all four available datasets:
42
+ - **RUSpellRU**: texts collected from ([LiveJournal](https://www.livejournal.com/media)), with manually corrected typos and errors;
43
+ - **MultidomainGold**: examples from 7 text sources, including the open web, news, social media, reviews, subtitles, policy documents and literary works;
44
+ - **MedSpellChecker**: texts with errors from medical anamnesis;
45
+ - **GitHubTypoCorpusRu**: spelling errors and typos in commits from [GitHub](https://github.com);
46
+
47
+ **RUSpellRU**
48
+ | Model | Precision | Recall | F1 |
49
+ | --- | --- | --- | --- |
50
+ | M2M100-1.2B | 59.4 | 43.3 | 50.1 |
51
+ | ChatGPT gpt-3.5-turbo-0301 | 55.8 | 75.3 | 64.1 |
52
+ | ChatGPT gpt-4-0314 | 57.0 | 75.9 | 63.9 |
53
+ | ChatGPT text-davinci-003 | 55.9 | 75.3 | 64.2 |
54
+ | Yandex.Speller | 83.0 | 59.8 | 69.5 |
55
+ | JamSpell | 42.1 | 32.8 | 36.9 |
56
+ | HunSpell | 31.3 | 34.9 | 33.0 |
57
+
58
+ **MultidomainGold**
59
+ | Model | Precision | Recall | F1 |
60
+ | --- | --- | --- | --- |
61
+ | M2M100-1.2B | 56.4 | 44.8 | 49.9 |
62
+ | ChatGPT gpt-3.5-turbo-0301 | 33.8 | 72.1 | 46.0 |
63
+ | ChatGPT gpt-4-0314 | 34.0 | 73.2 | 46.4 |
64
+ | ChatGPT text-davinci-003 | 33.6 | 72.0 | 45.8 |
65
+ | Yandex.Speller | 52.9 | 51.4 | 52.2 |
66
+ | JamSpell | 25.7 | 30.6 | 28.0 |
67
+ | HunSpell | 16.2 | 40.1 | 23.0 |
68
+
69
+ **MedSpellChecker**
70
+ | Model | Precision | Recall | F1 |
71
+ | --- | --- | --- | --- |
72
+ | M2M100-1.2B | 63.7 | 57.8 | 60.6 |
73
+ | ChatGPT gpt-3.5-turbo-0301 | 53.2 | 67.6 | 59.6 |
74
+ | ChatGPT gpt-4-0314 | 54.2 | 69.4 | 60.9 |
75
+ | ChatGPT text-davinci-003 | 47.8 | 68.4 | 56.3 |
76
+ | Yandex.Speller | 80.6 | 47.8 | 60.0 |
77
+ | JamSpell | 24.6 | 29.7 | 26.9 |
78
+ | HunSpell | 10.3 | 40.2 | 16.4 |
79
+
80
+ **GitHubTypoCorpusRu**
81
+ | Model | Precision | Recall | F1 |
82
+ | --- | --- | --- | --- |
83
+ | M2M100-1.2B | 45.7 | 41.4 | 43.5 |
84
+ | ChatGPT gpt-3.5-turbo-0301 | 43.8 | 57.0 | 49.6 |
85
+ | ChatGPT gpt-4-0314 | 45.2 | 58.2 | 51.0 |
86
+ | ChatGPT text-davinci-003 | 46.5 | 58.1 | 51.7 |
87
+ | Yandex.Speller | 67.7 | 37.5 | 48.3 |
88
+ | JamSpell | 49.5 | 29.9 | 37.3 |
89
+ | HunSpell | 28.5 | 30.7 | 29.6 |
90
+
91
+ ## How to use
92
+ ```python
93
+ from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
94
+ path_to_model = "ai-forever/RuM2M100-1.2B"
95
+ model = M2M100ForConditionalGeneration.from_pretrained(path_to_model)
96
+ tokenizer = M2M100Tokenizer.from_pretrained(path_to_model, src_lang="ru", tgt_lang="ru")
97
+ sentence = "прийдя в МГТУ я был удивлен никого необноружив там…"
98
+ encodings = tokenizer(sentence, return_tensors="pt")
99
+ generated_tokens = model.generate(
100
+ **encodings, forced_bos_token_id=tokenizer.get_lang_id("ru"))
101
+ answer = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
102
+ print(answer)
103
+ #["прийдя в МГТУ я был удивлен никого не обнаружив там..."]
104
+ ```
105
+
106
+ ## Resources
107
+ - [SAGE library](https://github.com/ai-forever/sage), GitHub
108
+ - [ruM2M100-1.2B](https://huggingface.co/ai-forever/RuM2M100-1.2B), HuggingFace
109
+ - [ruM2M100-418M](https://huggingface.co/ai-forever/RuM2M100-420M), HuggingFace
110
+ - [FredT5-large-spell](https://huggingface.co/ai-forever/FRED-T5-large-spell), HuggingFace
111
+ - [T5-large-spell](https://huggingface.co/ai-forever/T5-large-spell), HuggingFace
112
+
113
+ ## License
114
+ Model [M2M100-1.2B](https://huggingface.co/facebook/m2m100_1.2B), on the basis of which our solution is made, and its source code are supplied under the MIT open license.
115
+ Our solution also comes with MIT license.
116
+
117
+ ## Specifications
118
+ - File size: 5 Gb;
119
+ - Framework: pytorch
120
+ - Format: AI Service
121
+ - Version: v1.0
122
+ - Developer: SberDevices, AGI NLP
123
+
124
+ ## Contacts
125
+ [email protected]