tinybert-base-uncased-ADVQA36K-V1

This model is a fine-tuned version of deepset/tinybert-6l-768d-squad2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.9677

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 6
• eval_batch_size: 60
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
3.0765	0.0599	100	2.5814
2.8268	0.1198	200	2.5095
2.7793	0.1796	300	2.5010
2.7207	0.2395	400	2.4600
2.7237	0.2994	500	2.4738
2.7847	0.3593	600	2.4548
2.6382	0.4192	700	2.4683
2.581	0.4790	800	2.5506
2.6751	0.5389	900	2.4545
2.7498	0.5988	1000	2.3878
2.694	0.6587	1100	2.3868
2.4758	0.7186	1200	2.3428
2.6314	0.7784	1300	2.4042
2.6303	0.8383	1400	2.3783
2.5687	0.8982	1500	2.3797
2.5989	0.9581	1600	2.3255
2.207	1.0180	1700	2.7212
1.8539	1.0778	1800	2.3982
1.7812	1.1377	1900	2.6368
1.765	1.1976	2000	2.5626
1.7895	1.2575	2100	2.6103
1.8144	1.3174	2200	2.5863
1.7657	1.3772	2300	2.4383
1.8256	1.4371	2400	2.4881
1.8215	1.4970	2500	2.6227
1.9053	1.5569	2600	2.4121
1.872	1.6168	2700	2.4348
1.7558	1.6766	2800	2.4743
1.6951	1.7365	2900	2.6199
1.8006	1.7964	3000	2.5804
1.7538	1.8563	3100	2.4707
1.8206	1.9162	3200	2.4198
1.8098	1.9760	3300	2.4864
1.468	2.0359	3400	2.9562
1.3203	2.0958	3500	2.8562
1.2161	2.1557	3600	2.9124
1.2002	2.2156	3700	2.8811
1.1981	2.2754	3800	2.9929
1.151	2.3353	3900	3.0145
1.1867	2.3952	4000	2.9327
1.2305	2.4551	4100	2.9967
1.1003	2.5150	4200	3.0383
1.2409	2.5749	4300	2.9114
1.1997	2.6347	4400	2.9199
1.2403	2.6946	4500	2.8751
1.1495	2.7545	4600	2.9636
1.1879	2.8144	4700	2.9894
1.1685	2.8743	4800	2.9766
1.2527	2.9341	4900	2.9656
1.2254	2.9940	5000	2.9677

Framework versions

• Transformers 4.40.0
• Pytorch 2.6.0+cu124
• Datasets 2.21.0
• Tokenizers 0.19.1