ajiayi/bert-peft-singapore-digi-banks

DistilBERT PEFT (LoRA) for Singapore Bank Review Classification

This model is a Parameter-Efficient Fine-Tuned (PEFT) version of distilbert-base-uncased using LoRA (Low-Rank Adaptation) for 5-class star rating prediction (1-5 stars) on Singapore digital bank reviews.

🎯 Model Performance

Metric	Score
Test Accuracy	0.7619
Precision (Macro)	0.3064
Recall (Macro)	0.3597
F1-Score (Macro)	0.3307
MAE (Mean Absolute Error)	0.6000 stars

🚀 Quick Start

Loading the Model

from peft import PeftModel, PeftConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load configuration
config = PeftConfig.from_pretrained("ajiayi/bert-peft-singapore-digi-banks")

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    num_labels=5
)

# Load PEFT adapter
model = PeftModel.from_pretrained(base_model, "ajiayi/bert-peft-singapore-digi-banks")
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Make predictions
text = "Love this bank! Best interest rates in Singapore."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
outputs = model(**inputs)
predicted_stars = outputs.logits.argmax(dim=1).item() + 1  # Convert 0-4 to 1-5
print(f"Predicted rating: {predicted_stars} stars")

📊 Model Details

Model Architecture

• Base Model: distilbert-base-uncased (66M parameters)
• Fine-tuning Method: PEFT with LoRA
• Task: Multi-class text classification (5 classes)
• Classes: 1-star, 2-star, 3-star, 4-star, 5-star

LoRA Configuration

• Rank (r): 8
• Alpha: 16
• Dropout: 0.1
• Target Modules: q_lin, v_lin (attention query and value layers)

Efficiency Metrics

• Trainable Parameters: N/A
• Total Parameters: 67,846,666
• Trainable Percentage: N/A
• Adapter Size: ~5MB (vs ~250MB for full fine-tuning)
• Training Time: 0.96 minutes

🎓 Training Details

Training Data

• Domain: Singapore digital bank reviews (GXS, Trust Bank, MariBank, etc.)
• Training Samples: 489
• Validation Samples: 105
• Test Samples: 105
• Augmentation: LLM-based data augmentation using Google Gemini

Training Hyperparameters

• Learning Rate: 0.0002
• Batch Size: 8
• Gradient Accumulation Steps: 2
• Epochs: 3
• Weight Decay: 0.01
• Warmup Ratio: 0.0
• Optimizer: AdamW
• Loss Function: Cross-Entropy with class weights

Training Infrastructure

• Framework: HuggingFace Transformers + PEFT
• Hardware: Apple Silicon (MPS) / CPU
• Mixed Precision: FP16 (when GPU available)

🎯 Intended Use

Primary Use Cases

• Sentiment analysis of Singapore digi bank reviews
• Star rating prediction for financial services
• Customer feedback classification
• Banking app review analysis

Out-of-Scope Use

• Reviews for non-Singapore banks (model trained specifically on Singapore digital banks)
• General sentiment analysis (domain-specific model)

⚖️ Limitations and Bias

Known Limitations

• Trained on Singapore digital bank reviews only
• May not generalize well to traditional banks
• Performance may vary on very short or very long reviews

Potential Biases

• Data collected from Apple App Store only
• May reflect platform-specific sentiment patterns
• Class imbalance handled via weighted loss and augmentation (LLM-generated data augmented minority classes)

📈 Evaluation Results

The model was evaluated on a held-out test set with the following results:

• Accuracy: 76.19%
• Average prediction error: 0.60 stars

🔧 How to Use

For Inference Only

from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=5)
model = PeftModel.from_pretrained(model, "ajiayi/bert-peft-singapore-digi-banks")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

# Predict
def predict_rating(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    model.eval()
    with torch.no_grad():
        outputs = model(**inputs)
        prediction = outputs.logits.argmax(dim=1).item()
    return prediction + 1  # Convert 0-indexed to 1-5 stars

# Example
review = "Great banking app! Very user-friendly interface."
stars = predict_rating(review)
print(f"Predicted: {stars} stars")

📚 Citation

@misc{peft-singapore-banks,
  author = {Your Name},
  title = {PEFT DistilBERT for Singapore Bank Review Classification},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/ajiayi/bert-peft-singapore-digi-banks}}
}

📄 License

This model is released under the Apache 2.0 License.

🙏 Acknowledgments

• Base model: DistilBERT by HuggingFace
• PEFT library by HuggingFace
• Training framework: HuggingFace Transformers

---

Generated: 2025-10-14 23:08:12