Hugging Face's logo

gauneg
/
deberta-v3-large-absa-ate-sentiment-lora-adapter

Token Classification
Transformers
Safetensors
English
LoRA
Adapter
Model card Files
xet
Community
gauneg commited on
Commit
a0da3fe
·
verified ·
1 Parent(s): 5cb37a2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +197 -3
README.md CHANGED
@@ -1,3 +1,197 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ base_model:
6
+ - microsoft/deberta-v3-large
7
+ pipeline_tag: token-classification
8
+ library_name: transformers
9
+ tags:
10
+ - LoRA
11
+ - Adapter
12
+ ---
13
+ # Training
14
+ This model adapter is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms.
15
+ The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed.
16
+ It has been created using [PEFT](https://huggingface.co/docs/peft/index) framework for [LoRA:Low-Rank Adaptation](https://arxiv.org/abs/2106.09685).
17
+
18
+ ## Datasets
19
+ This model has been trained on the following datasets:
20
+
21
+ 1. Aspect Based Sentiment Analysis SemEval Shared Tasks ([2014](https://aclanthology.org/S14-2004/), [2015](https://aclanthology.org/S15-2082/), [2016](https://aclanthology.org/S16-1002/))
22
+ 2. Multi-Aspect Multi-Sentiment [MAMS](https://aclanthology.org/D19-1654/)
23
+
24
+ # Use
25
+
26
+ * Loading the base model and merging it with LoRA parameters
27
+
28
+ ```python
29
+ from transformers import AutoTokenizer, AutoModelForTokenClassification
30
+ from peft import PeftModel
31
+
32
+ # preparing the labels
33
+ labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
34
+ id2labels = {k:lab for lab, k in labels.items()}
35
+ labels2ids = {k:lab for lab, k in id2labels.items()}
36
+
37
+ # loading tokenizer and base_model
38
+ base_id = 'microsoft/deberta-v3-large'
39
+ tokenizer = AutoTokenizer.from_pretrained(base_id)
40
+ base_model = AutoModelForTokenClassification.from_pretrained(base_id, num_labels=len(labels), id2label=id2labels, label2id=labels2ids)
41
+
42
+ # using this adapter with base model
43
+ model = PeftModel.from_pretrained(base_model, 'gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter', is_trainable=False)
44
+ ```
45
+
46
+ This model can be utilized in the following two methods:
47
+
48
+ 1. Using pipelines for end to end inference
49
+ 2. Making token level inference
50
+
51
+
52
+ # Using end-to-end token classification pipeline
53
+ ```python
54
+ from transformers import pipeline
55
+ ate_sent_pipeline = pipeline(task='ner',
56
+ aggregation_strategy='simple',
57
+ model=model,
58
+ tokenizer=tokenizer)
59
+
60
+ text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
61
+ ate_sent_pipeline(text_input)
62
+ ```
63
+ Expected output
64
+
65
+ ```bash
66
+ [{'entity_group': 'pos',
67
+ 'score': 0.89315575,
68
+ 'word': 'food',
69
+ 'start': 25,
70
+ 'end': 30},
71
+ {'entity_group': 'neg',
72
+ 'score': 0.9100719,
73
+ 'word': 'service',
74
+ 'start': 55,
75
+ 'end': 63}]
76
+ ```
77
+ # OR
78
+
79
+ # Making token level inference
80
+
81
+ ```python
82
+ # making one prediction at a time (should be padded/batched and truncated for efficiency)
83
+ text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
84
+ tok_inputs = tokenizer(text_input, return_tensors="pt")
85
+
86
+
87
+ y_pred = model(**tok_inputs) # predicting the logits
88
+
89
+ # since first and the last tokens are excluded ([CLS] and [SEP]) they have to be removed before decoding
90
+ y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits
91
+
92
+
93
+ decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin]
94
+
95
+
96
+ ## displaying the input tokens with predictions and skipping [CLS] and [SEP] tokens at the beginning and the end respectively
97
+ decoded_toks = tok_inputs['input_ids'][0][1:-1]
98
+ tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))
99
+ ```
100
+ Expected Results
101
+
102
+ ```bash
103
+ [('▁Been', 'O'),
104
+ ('▁here', 'O'),
105
+ ('▁a', 'O'),
106
+ ('▁few', 'O'),
107
+ ('▁times', 'O'),
108
+ ('▁and', 'O'),
109
+ ('▁food', 'B-pos'),
110
+ ('▁has', 'O'),
111
+ ('▁always', 'O'),
112
+ ('▁been', 'O'),
113
+ ('▁good', 'O'),
114
+ ('▁but', 'O'),
115
+ ('▁service', 'B-neg'),
116
+ ('▁really', 'O'),
117
+ ('▁suffers', 'O'),
118
+ ('▁when', 'O'),
119
+ ('▁it', 'O'),
120
+ ('▁gets', 'O'),
121
+ ('▁crowded', 'O'),
122
+ ('.', 'O')]
123
+ ```
124
+
125
+ # Evaluation on Benchmark Test Datasets
126
+
127
+ The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
128
+ on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)
129
+
130
+ # ATE (Aspect Term Extraction Only)
131
+ | Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
132
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
133
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.16|73.92|71.6|
134
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|70.92|72.28|71.07|
135
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|64.05|79.69|70.0|
136
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.29|72.78|68.92|
137
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
138
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|70.58|61.52|64.21|
139
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|66.38|50.62|54.31|
140
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|70.82|48.97|52.08|
141
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|73.61|46.38|49.87|
142
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
143
+ |MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|81.07|79.66|80.35|
144
+ |MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|79.91|78.95|79.39|
145
+ |MAMS-ATE (2019)|microsoft/deberta-v3-large|(this)[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|74.46|84.5|78.75|
146
+ |MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|77.8|79.81|78.75|
147
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
148
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|88.59|87.0|87.45|
149
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|92.26|82.95|86.57|
150
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|93.07|81.95|86.32|
151
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|92.94|81.71|86.01|
152
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
153
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.91|75.4|72.74|
154
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.54|77.48|72.63|
155
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|68.32|79.84|72.28|
156
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.94|74.75|71.84|
157
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
158
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|70.22|75.83|71.84|
159
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|71.54|73.38|71.2|
160
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|71.35|72.78|70.85|
161
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|(this)[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|66.68|77.97|70.79|
162
+
163
+ # Aspect Sentiment Evaluation
164
+ This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens
165
+ on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)
166
+ | Test Dataset | Base Model | Fine-tuned Model | Precision | Recall | F1 Score |
167
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
168
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.92|65.55|54.94|
169
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.62|53.65|54.08|
170
+ |hotel reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|55.43|56.53|54.03|
171
+ |hotel reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|52.88|55.19|53.85|
172
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
173
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|44.25|41.55|42.81|
174
+ |laptop reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|46.15|33.23|37.09|
175
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|41.7|34.38|36.93|
176
+ |laptop reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|44.98|31.87|35.67|
177
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
178
+ |MAMS-ATE (2019)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|72.06|72.98|72.49|
179
+ |MAMS-ATE (2019)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|72.97|71.63|72.26|
180
+ |MAMS-ATE (2019)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|69.34|73.3|71.07|
181
+ |MAMS-ATE (2019)|microsoft/deberta-v3-large|(this)[gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|65.74|75.11|69.77|
182
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
183
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|61.15|58.46|59.74|
184
+ |restaurant reviews (SemEval 2014)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|60.13|56.81|58.13|
185
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|56.79|59.3|57.93|
186
+ |restaurant reviews (SemEval 2014)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|58.99|54.76|56.45|
187
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
188
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.89|55.7|54.11|
189
+ |restaurant reviews (SemEval 2015)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|54.36|55.38|53.6|
190
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|51.67|56.58|53.29|
191
+ |restaurant reviews (SemEval 2015)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|54.55|53.68|53.12|
192
+ | ------------ | ---------- | ---------------- | --------- | ------ | -------- |
193
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-large|[gauneg/roberta-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/roberta-large-absa-ate-sentiment-lora-adapter)|53.7|60.49|55.05|
194
+ |restaurant reviews (SemEval 2016)|FacebookAI/roberta-base|[gauneg/roberta-base-absa-ate-sentiment](https://huggingface.co/gauneg/roberta-base-absa-ate-sentiment)|52.31|54.58|52.33|
195
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-base|[gauneg/deberta-v3-base-absa-ate-sentiment](https://huggingface.co/gauneg/deberta-v3-base-absa-ate-sentiment)|52.07|54.58|52.15|
196
+ |restaurant reviews (SemEval 2016)|microsoft/deberta-v3-large|(this) [gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter](https://huggingface.co/gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter)|49.07|56.5|51.25|
197
+