Hugging Face's logo

aiface
/
CafeBERT_massive_v3

Transformers
Safetensors
Generated from Trainer
Model card Files
xet
Community
aiface commited on
Commit
2e366d1
·
verified ·
1 Parent(s): 0afc7d3

Model save

Browse files
Files changed (5) hide show
  1. README.md +81 -0
  2. intent_report_test.txt +75 -0
  3. model.safetensors +1 -1
  4. model_predict_test.csv +0 -0
  5. slot_report_test.txt +60 -0
README.md ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: uitnlp/CafeBERT
5
+ tags:
6
+ - generated_from_trainer
7
+ model-index:
8
+ - name: CafeBERT_massive_v3
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # CafeBERT_massive_v3
16
+
17
+ This model is a fine-tuned version of [uitnlp/CafeBERT](https://huggingface.co/uitnlp/CafeBERT) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 5.2297
20
+ - Slot P: 0.7256
21
+ - Slot R: 0.7801
22
+ - Slot F1: 0.7519
23
+ - Slot Exact Match: 0.7260
24
+ - Intent Acc: 0.8633
25
+
26
+ ## Model description
27
+
28
+ More information needed
29
+
30
+ ## Intended uses & limitations
31
+
32
+ More information needed
33
+
34
+ ## Training and evaluation data
35
+
36
+ More information needed
37
+
38
+ ## Training procedure
39
+
40
+ ### Training hyperparameters
41
+
42
+ The following hyperparameters were used during training:
43
+ - learning_rate: 5e-05
44
+ - train_batch_size: 128
45
+ - eval_batch_size: 128
46
+ - seed: 42
47
+ - gradient_accumulation_steps: 2
48
+ - total_train_batch_size: 256
49
+ - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
50
+ - lr_scheduler_type: cosine
51
+ - lr_scheduler_warmup_ratio: 0.06
52
+ - num_epochs: 30
53
+ - mixed_precision_training: Native AMP
54
+
55
+ ### Training results
56
+
57
+ | Training Loss | Epoch | Step | Validation Loss | Slot P | Slot R | Slot F1 | Slot Exact Match | Intent Acc |
58
+ |:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:-------:|:----------------:|:----------:|
59
+ | No log | 1.0 | 45 | 16.4744 | 0.2572 | 0.0801 | 0.1222 | 0.3114 | 0.0930 |
60
+ | 72.336 | 2.0 | 90 | 6.6272 | 0.5583 | 0.6075 | 0.5818 | 0.5268 | 0.6021 |
61
+ | 20.704 | 3.0 | 135 | 4.4083 | 0.6449 | 0.7338 | 0.6865 | 0.6596 | 0.7885 |
62
+ | 9.4202 | 4.0 | 180 | 3.7185 | 0.6867 | 0.7493 | 0.7166 | 0.6926 | 0.8367 |
63
+ | 6.3529 | 5.0 | 225 | 3.6662 | 0.7024 | 0.7726 | 0.7358 | 0.7083 | 0.8647 |
64
+ | 4.7787 | 6.0 | 270 | 3.8379 | 0.7102 | 0.7657 | 0.7369 | 0.7122 | 0.8667 |
65
+ | 3.6929 | 7.0 | 315 | 3.7687 | 0.7152 | 0.7796 | 0.7460 | 0.7191 | 0.8652 |
66
+ | 2.9663 | 8.0 | 360 | 4.1024 | 0.7037 | 0.7905 | 0.7446 | 0.7186 | 0.8677 |
67
+ | 2.4189 | 9.0 | 405 | 4.2478 | 0.7177 | 0.7856 | 0.7501 | 0.7206 | 0.8692 |
68
+ | 1.9492 | 10.0 | 450 | 4.4022 | 0.7179 | 0.7801 | 0.7477 | 0.7275 | 0.8697 |
69
+ | 1.9492 | 11.0 | 495 | 4.6437 | 0.7095 | 0.7751 | 0.7408 | 0.7122 | 0.8706 |
70
+ | 1.5099 | 12.0 | 540 | 4.7049 | 0.7223 | 0.7881 | 0.7537 | 0.7290 | 0.8706 |
71
+ | 1.2818 | 13.0 | 585 | 4.9417 | 0.7189 | 0.7811 | 0.7487 | 0.7245 | 0.8677 |
72
+ | 1.0546 | 14.0 | 630 | 5.0501 | 0.7188 | 0.7781 | 0.7473 | 0.7226 | 0.8667 |
73
+ | 0.8641 | 15.0 | 675 | 5.2297 | 0.7256 | 0.7801 | 0.7519 | 0.7260 | 0.8633 |
74
+
75
+
76
+ ### Framework versions
77
+
78
+ - Transformers 4.55.0
79
+ - Pytorch 2.7.0+cu126
80
+ - Datasets 3.6.0
81
+ - Tokenizers 0.21.4
intent_report_test.txt ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ precision recall f1-score support
2
+
3
+ 0 0.92 0.97 0.94 88
4
+ 1 0.89 0.89 0.89 36
5
+ 2 0.97 0.94 0.96 35
6
+ 3 0.94 0.83 0.88 35
7
+ 4 0.83 0.92 0.87 26
8
+ 5 0.00 0.00 0.00 1
9
+ 6 0.79 0.70 0.74 43
10
+ 7 1.00 0.50 0.67 4
11
+ 8 0.95 1.00 0.97 18
12
+ 9 0.94 0.90 0.92 72
13
+ 10 0.97 1.00 0.99 39
14
+ 11 0.79 1.00 0.88 15
15
+ 12 0.71 0.57 0.63 169
16
+ 13 0.94 0.97 0.95 156
17
+ 14 0.86 0.92 0.89 13
18
+ 15 0.60 0.75 0.67 12
19
+ 16 0.80 0.91 0.85 22
20
+ 17 0.66 0.81 0.72 26
21
+ 18 0.86 0.93 0.89 27
22
+ 19 0.76 0.94 0.84 31
23
+ 20 0.90 0.88 0.89 41
24
+ 21 0.86 0.92 0.89 39
25
+ 22 0.83 0.85 0.84 124
26
+ 23 1.00 0.88 0.94 34
27
+ 24 1.00 0.90 0.95 10
28
+ 25 0.95 1.00 0.97 19
29
+ 26 0.92 0.84 0.88 57
30
+ 27 0.91 0.80 0.85 25
31
+ 28 0.40 0.33 0.36 6
32
+ 29 1.00 0.50 0.67 6
33
+ 30 0.93 0.96 0.94 67
34
+ 31 0.95 0.86 0.90 21
35
+ 32 0.65 0.83 0.73 126
36
+ 33 0.97 0.92 0.95 114
37
+ 34 1.00 0.92 0.96 26
38
+ 35 0.85 1.00 0.92 11
39
+ 36 0.82 0.94 0.88 72
40
+ 37 0.00 0.00 0.00 0
41
+ 38 0.91 0.67 0.77 15
42
+ 39 0.88 0.84 0.86 25
43
+ 40 1.00 0.98 0.99 43
44
+ 41 0.50 0.33 0.40 3
45
+ 42 0.87 0.88 0.87 51
46
+ 43 0.94 0.89 0.91 36
47
+ 44 0.98 0.95 0.97 119
48
+ 45 0.92 0.93 0.92 176
49
+ 46 0.94 0.94 0.94 32
50
+ 47 0.99 0.93 0.96 81
51
+ 48 0.93 0.95 0.94 41
52
+ 49 0.81 0.81 0.81 141
53
+ 50 0.91 0.92 0.91 209
54
+ 51 0.97 0.91 0.94 35
55
+ 52 1.00 1.00 1.00 21
56
+ 53 0.94 0.92 0.93 52
57
+ 54 0.96 1.00 0.98 23
58
+ 55 0.70 0.80 0.74 20
59
+ 56 0.97 1.00 0.99 36
60
+ 57 0.91 0.86 0.88 35
61
+ 58 0.90 0.84 0.87 63
62
+ 59 0.85 0.78 0.82 51
63
+
64
+ accuracy 0.88 2974
65
+ macro avg 0.85 0.83 0.83 2974
66
+ weighted avg 0.88 0.88 0.88 2974
67
+
68
+ Confusion matrix:
69
+ [[85 0 0 ... 0 0 0]
70
+ [ 0 32 0 ... 0 0 0]
71
+ [ 0 0 33 ... 0 0 0]
72
+ ...
73
+ [ 0 0 0 ... 30 0 0]
74
+ [ 0 0 0 ... 0 53 0]
75
+ [ 0 0 0 ... 0 1 40]]
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f624df326ea9f072e9eed63e0a11eec879fbbff0c50f9e9f6ab184ecc3e24a53
3
  size 2240362200
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b90309496f45eaee31293ecd7fce2d94d04143a9e31bb5e9abf993e39833a7a3
3
  size 2240362200
model_predict_test.csv ADDED
The diff for this file is too large to render. See raw diff
 
slot_report_test.txt ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ precision recall f1-score support
2
+
3
+ alarm_type 0.67 1.00 0.80 2
4
+ app_name 0.42 1.00 0.59 5
5
+ artist_name 0.76 0.82 0.79 61
6
+ audiobook_author 0.67 0.80 0.73 5
7
+ audiobook_name 0.77 0.74 0.76 23
8
+ business_name 0.84 0.88 0.86 92
9
+ business_type 0.47 0.55 0.51 31
10
+ change_amount 0.40 0.44 0.42 9
11
+ coffee_type 0.60 0.75 0.67 4
12
+ color_type 0.64 0.81 0.71 26
13
+ cooking_type 0.56 0.62 0.59 8
14
+ currency_name 0.83 0.96 0.89 50
15
+ date 0.81 0.87 0.84 415
16
+ definition_word 0.84 0.80 0.82 51
17
+ device_type 0.68 0.79 0.73 57
18
+ drink_type 0.00 0.00 0.00 1
19
+ email_address 1.00 1.00 1.00 9
20
+ email_folder 0.57 0.80 0.67 5
21
+ event_name 0.60 0.70 0.65 260
22
+ food_type 0.62 0.71 0.66 72
23
+ game_name 0.92 0.88 0.90 26
24
+ general_frequency 0.58 0.70 0.64 20
25
+ house_place 0.84 0.91 0.88 58
26
+ ingredient 0.17 0.17 0.17 6
27
+ joke_type 0.75 0.82 0.78 11
28
+ list_name 0.76 0.79 0.77 61
29
+ meal_type 0.53 0.89 0.67 18
30
+ media_type 0.84 0.84 0.84 128
31
+ movie_name 0.33 0.50 0.40 2
32
+ movie_type 1.00 0.67 0.80 3
33
+ music_album 0.00 0.00 0.00 1
34
+ music_descriptor 0.20 0.14 0.17 7
35
+ music_genre 0.65 0.82 0.73 50
36
+ news_topic 0.47 0.54 0.50 52
37
+ order_type 0.68 0.85 0.76 20
38
+ person 0.76 0.88 0.81 216
39
+ personal_info 0.50 0.79 0.61 14
40
+ place_name 0.86 0.81 0.83 281
41
+ player_setting 0.55 0.40 0.46 40
42
+ playlist_name 0.33 0.20 0.25 15
43
+ podcast_descriptor 0.61 0.46 0.52 24
44
+ podcast_name 0.67 0.59 0.62 17
45
+ radio_name 0.55 0.67 0.60 33
46
+ relation 0.62 0.76 0.69 59
47
+ song_name 0.64 0.82 0.72 39
48
+ sport_type 0.00 0.00 0.00 0
49
+ time 0.70 0.72 0.71 191
50
+ time_zone 0.64 0.69 0.67 13
51
+ timeofday 0.67 0.73 0.70 60
52
+ transport_agency 0.90 1.00 0.95 9
53
+ transport_descriptor 0.00 0.00 0.00 2
54
+ transport_name 1.00 0.75 0.86 4
55
+ transport_type 0.81 0.86 0.84 65
56
+ weather_descriptor 0.60 0.71 0.65 82
57
+
58
+ micro avg 0.72 0.78 0.75 2813
59
+ macro avg 0.61 0.67 0.63 2813
60
+ weighted avg 0.72 0.78 0.75 2813