|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- qwen |
|
|
- unsloth |
|
|
- cybersecurity |
|
|
- instruction-tuning |
|
|
- lora |
|
|
- kaggle |
|
|
base_model: unsloth/Qwen3-0.6B |
|
|
datasets: |
|
|
- zobayer0x01/cybersecurity-qa |
|
|
metrics: |
|
|
- sacrebleu |
|
|
- rougeL |
|
|
- chrf++ |
|
|
- bertscore |
|
|
- perplexity |
|
|
--- |
|
|
|
|
|
# qwen3-0-6b — Cybersecurity QA (LORA) |
|
|
Fine-tuned on Kaggle using **LORA**. |
|
|
|
|
|
|
|
|
### Model Summary |
|
|
- Base: `unsloth/Qwen3-0.6B` |
|
|
- Trainable params: **10,092,544** / total **606,142,464** |
|
|
- Train wall time (s): 20817.1 |
|
|
- Files: adapter_model.safetensors + adapter_config.json (LoRA) + tokenizer files |
|
|
|
|
|
### Data |
|
|
- Dataset: `zobayer0x01/cybersecurity-qa` |
|
|
- Samples: **total=42427**, train=38184, val=1200 |
|
|
- Prompting: Chat template with a fixed system prompt: |
|
|
```text |
|
|
You are a helpful assistant specialized in cybersecurity Q&A. |
|
|
``` |
|
|
|
|
|
|
|
|
### Training Config |
|
|
| Field | Value | |
|
|
|---|---| |
|
|
| Method | **LORA** | |
|
|
| Precision | fp16 | |
|
|
| Quantization | none | |
|
|
| Mode | epochs | |
|
|
| Num Epochs | 1 | |
|
|
| Max Steps | 5000 | |
|
|
| Eval Steps | 500 | |
|
|
| Save Steps | 500 | |
|
|
| LR | 0.0001 | |
|
|
| Max Length | 768 | |
|
|
| per_device_batch_size | 1 | |
|
|
| grad_accum | 8 | |
|
|
|
|
|
### Evaluation (greedy, fixed-length decode) |
|
|
| Metric | Score | |
|
|
|---|---:| |
|
|
| BLEU-4 | 1.33 | |
|
|
| ROUGE-L | 13.31 | |
|
|
| F1 (token-level) | 25.59 | |
|
|
| chrF++ | 20.07 | |
|
|
| BERTScore F1 | 82.60 | |
|
|
| Perplexity | 16.29 | |
|
|
|
|
|
> Notes: We normalize whitespace/punctuations, compute token-level P/R/F1, and use `evaluate`'s `sacrebleu/rouge/chrf/bertscore`. |
|
|
|
|
|
## How to use |
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
from peft import PeftModel |
|
|
tok = AutoTokenizer.from_pretrained("nhonhoccode/qwen3-0-6b-cybersecqa-lora-20251112-1035") |
|
|
base = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3-0.6B") |
|
|
mdl = PeftModel.from_pretrained(base, "nhonhoccode/qwen3-0-6b-cybersecqa-lora-20251112-1035") # Loads LoRA adapter |
|
|
prompt = tok.apply_chat_template( |
|
|
[{"role":"system","content":"You are a helpful assistant specialized in cybersecurity Q&A."}, |
|
|
{"role":"user","content":"Explain SQL injection in one paragraph."}], |
|
|
tokenize=False, add_generation_prompt=True |
|
|
) |
|
|
ids = tok(prompt, return_tensors="pt").input_ids |
|
|
out = mdl.generate(ids, max_new_tokens=128, do_sample=False) |
|
|
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
### Intended Use & Limitations |
|
|
- Domain: cybersecurity Q&A; not guaranteed to be accurate for legal/medical purposes. |
|
|
- The model can hallucinate or produce outdated guidance—verify before applying in production. |
|
|
- Safety: No explicit content filtering. Add guardrails (moderation, retrieval augmentation) for deployment. |
|
|
|
|
|
### Reproducibility (env) |
|
|
- `transformers>=4.43,<5`, `accelerate>=0.33,<0.34`, `peft>=0.11,<0.13`, `datasets>=2.18,<3`, `evaluate>=0.4,<0.5`, |
|
|
`rouge-score`, `sacrebleu`, `huggingface_hub>=0.23,<0.26`, `bitsandbytes` |
|
|
- GPU: T4-class; LoRA recommended for low VRAM. |
|
|
|
|
|
### Changelog |
|
|
- 2025-11-12 10:36 — Initial release (LORA) |
|
|
|
