Create README.md

README.md

@@ -0,0 +1,153 @@
+---
+license: mit
+language:
+- en
+metrics:
+- f1
+- accuracy
+base_model:
+- google-bert/bert-base-uncased
+pipeline_tag: text-classification
+library_name: transformers
+tags:
+- hate-speech-detection,
+- explainability
+- attention
+- interpretable -ml
+- AAAI2026
+---
+
+# SRA-BERT for Hate Speech Detection
+
+**Supervised Rational Attention (SRA)** fine-tuned BERT model for explainable hate speech detection.
+
+## Abstract
+
+The opaque nature of deep learning models presents significant challenges for the ethical deployment of hate speech detection systems. To address this limitation, we introduce Supervised Rational Attention (SRA), a framework that explicitly aligns model attention with human rationales, improving both interpretability and fairness in hate speech classification. SRA integrates a supervised attention mechanism into transformer-based classifiers, optimizing a joint objective that combines standard classification loss with an alignment loss term that minimizes the discrepancy between attention weights and human-annotated rationales. Empirically, SRA achieves **2.4× better explainability** compared to current baselines, and produces token-level explanations that are more faithful and human-aligned.
+
+📄 [Read the paper on arXiv](https://arxiv.org/abs/2511.07065)
+
+📄 **Paper:** *"Aligning Attention with Human Rationales for Self-Explaining Hate Speech Detection"*  
+🎯 **Accepted:** AAAI-26 AI Alignment Track  
+🔗 **Demo:** [Live Demo on HuggingFace Spaces](https://huggingface.co/spaces/bragee/sra-hate-speech-demo)
+
+## Model Description
+
+This model is a `bert-base-uncased` classifier fine-tuned on the [HateXplain](https://github.com/hate-alert/HateXplain) dataset with **Supervised Rational Attention (SRA)** — a training method that aligns the model's attention weights with human-annotated rationales.
+
+### Key Innovation
+
+Standard BERT attention weights don't reliably explain predictions. SRA supervises a specific attention head (Layer 8, Head 7) to attend to the same tokens that human annotators identified as evidence for their labeling decisions.
+
+**Result:** Attention weights that actually explain the model's decisions, achieving **2.4× better alignment** with human rationales while maintaining classification performance.
+
+## Labels
+
+| Label | ID | Description |
+|-------|-----|-------------|
+| Normal | 0 | Non-hateful, non-offensive content |
+| Offensive | 1 | Offensive but not hate speech |
+| Hate Speech | 2 | Hate speech targeting protected groups |
+
+## Usage
+```python
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+import torch
+
+model_name = "bragee/sra-hate-speech-bert"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+
+text = "I love spending time with my friends"
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
+
+with torch.no_grad():
+    outputs = model(**inputs)
+    probs = torch.softmax(outputs.logits, dim=1)
+    pred = torch.argmax(probs, dim=1).item()
+
+labels = ["Normal", "Offensive", "Hate Speech"]
+print(f"Prediction: {labels[pred]} ({probs[0][pred]:.1%})")
+```
+
+### Extracting Attention-Based Explanations
+
+The key feature of this model is interpretable attention. Extract attention from the supervised head:
+```python
+with torch.no_grad():
+    outputs = model(**inputs, output_attentions=True)
+
+# Extract attention from Layer 8, Head 7 (the supervised head)
+attention = outputs.attentions[8][:, 7, :, :]  # (batch, seq, seq)
+attention_weights = attention.mean(dim=1)  # Average over query positions
+
+tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
+for token, weight in zip(tokens, attention_weights[0]):
+    if token not in ["[CLS]", "[SEP]", "[PAD]"]:
+        print(f"{token}: {weight:.3f}")
+```
+
+## Training Details
+
+### Training Data
+
+- **Dataset:** [HateXplain](https://github.com/hate-alert/HateXplain) 
+- **Size:** ~20,000 posts from Twitter and Gab
+- **Annotations:** 3-class labels + token-level rationales from multiple annotators
+
+### Training Procedure
+
+- **Base model:** `bert-base-uncased`
+- **Epochs:** 5
+- **Batch size:** 16
+- **Learning rate:** 2e-5
+- **Max sequence length:** 128
+
+### SRA Configuration
+
+- **Supervised attention head:** Layer 8, Head 7
+- **Attention loss weight (α):** 10.0
+- **Loss function:** Cross-entropy + α × MSE(attention, rationale)
+
+The MSE loss is only computed for offensive/hate speech examples where human rationales exist.
+
+## Evaluation
+
+Evaluated on the HateXplain test set:
+
+| Metric | Score |
+|--------|-------|
+| Macro F1 | 0.68 |
+| Accuracy | 0.70 |
+| Attention-Rationale Alignment | 2.4× baseline |
+
+## Intended Use
+
+- **Primary use:** Research on explainable AI and hate speech detection
+- **Demo/educational:** Understanding how attention-based explanations work
+- **Content moderation research:** Studying interpretable classifiers
+
+### Limitations
+
+- Trained on English Twitter/Gab data — may not generalize to other platforms or languages
+- Attention explanations are post-hoc interpretations, not guaranteed causal explanations
+- Should not be used as sole arbiter for content moderation decisions
+
+### Ethical Considerations
+
+This model is intended for research purposes. Hate speech detection systems can have false positives that disproportionately affect marginalized groups. Human review should always be part of any content moderation pipeline.
+
+## Citation
+```bibtex
+@article{eilertsen2025sra,
+  title={Aligning Attention with Human Rationales for Self-Explaining Hate Speech Detection},
+  author={Eilertsen, Brage and Bjørgfinsdóttir, Røskva and Vargas, Francielle and Ramezani-Kebrya, Ali},
+  journal={arXiv preprint arXiv:2511.07065},
+  year={2025},
+  note={Accepted at AAAI-26}
+}
+```
+
+## Model Card Contact
+
+For questions about this model, please open an issue on the [model repository](https://huggingface.co/bragee/sra-hate-speech-bert).