Model save

README.md

@@ -1,137 +1,51 @@
 ---
+library_name: transformers
 license: apache-2.0
-datasets:
-- yentinglin/TaiwanChat
-language:
-- zh
-base_model:
-- HuggingFaceTB/SmolLM2-135M-Instruct
-pipeline_tag: text-generation
+base_model: HuggingFaceTB/SmolLM2-135M-Instruct
+tags:
+- generated_from_trainer
+model-index:
+- name: SmolLM2-135M-Instruct-TaiwanChat
+  results: []
 ---
-# SmolLM2‑135M‑Instruct‑TaiwanChat
 
-A fine‑tuned SmolLM2‑135M Instruct model on the TaiwanChat dataset, optimized for multi‑turn Traditional Chinese conversational AI, using Unsloth 2025.3.19 with 4‑bit quantization and LoRA adapters.
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
----
-## Model Description
-
-- **Base model:** `HuggingFaceTB/SmolLM2-135M-Instruct`
-- **Fine‑tuned on:** `yentinglin/TaiwanChat` (subset of 85,840 examples ≈ 20 M tokens)
-- **Task:** Instruction‑tuned chat in Mandarin/Taiwanese
-- **Framework:** Unsloth + Hugging Face Transformers [`Trainer`] + PEFT (LoRA)
-- **Precision:**
-  - 4‑bit quantization on model weights
-  - FP16 on CUDA (V100)
-  - BF16 on Intel XPU (if available)
-- **Adapters:** LoRA (r=8, α=16) applied to `q_proj` and `v_proj` layers
-- **Memory optimizations:**
-  - Gradient checkpointing enabled
-  - CPU offload via DeepSpeed ZeRO Stage 2 (optional)
-
----
-## How to Use
-
-### 1. Install dependencies
-```bash
-pip install transformers datasets accelerate unsloth peft wandb
-# (optional) pip install xformers deepspeed
-```
-
-### 2. Load & Generate
-```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
-
-model_id = "Luigi/SmolLM2-135M-Instruct-TaiwanChat"
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/pesi/SmolLM2-135M-Instruct-TaiwanChat/runs/oy14fkq9)
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/pesi/SmolLM2-135M-Instruct-TaiwanChat/runs/oy14fkq9)
+# SmolLM2-135M-Instruct-TaiwanChat
 
-device = "cpu"
-if torch.cuda.is_available():
-    device = "cuda"
-elif torch.xpu.is_available():
-    device = "xpu"
+This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) on an unknown dataset.
 
-# Load
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model     = AutoModelForCausalLM.from_pretrained(model_id).to(device)
+## Model description
 
-# Inference pipeline
-generator = pipeline(
-    "text-generation",
-    model=model,
-    tokenizer=tokenizer,
-    device=0 if device in ("cuda","xpu") else -1,
-    max_new_tokens=512,
-    do_sample=True,
-    temperature=0.8,
-)
+More information needed
 
-prompt = "請問台北今天的天氣如何？"
-result = generator(prompt)
-print(result[0]["generated_text"])
-```
+## Intended uses & limitations
 
----
-## Training Script
-
-All training logic is contained in `train_with_unsloth.py`.
+More information needed
 
-**Key settings** (top of script):
-```python
-PROJECT_NAME = 'SmolLM2-135M-Instruct-TaiwanChat'
-BASE_MODEL_ID = 'HuggingFaceTB/SmolLM2-135M-Instruct'
-DATASET_ID   = 'yentinglin/TaiwanChat'
-N_SAMPLES    = 85840       # ~20M tokens subset
-MAX_LEN      = 256
-```
+## Training and evaluation data
 
-**PEFT & Quantization**:
-- Load in 4‑bit via Unsloth’s `FastLanguageModel.from_pretrained(..., load_in_4bit=True)`
-- Prepare for k‑bit training (`prepare_model_for_kbit_training`)
-- Attach LoRA adapters (r=8, α=16) to `q_proj`, `v_proj`
-- Enable gradient checkpointing on the model
+More information needed
 
-**Trainer hyperparameters**:
-- `per_device_train_batch_size = 1`
-- `gradient_accumulation_steps = 16`
-- `learning_rate = 5e-5`
-- `num_train_epochs = 3`
-- `fp16` on CUDA, `bf16` on XPU
-- `logging_steps = 1000`
-- `save_steps = 5000`
-- `gradient_checkpointing = True`
-- `push_to_hub = True`
+## Training procedure
 
-### Run training
-```bash
-python train_with_unsloth.py
-```
+### Training hyperparameters
 
-The script will:
-1. Auto‑detect **CUDA**, **XPU**, or **CPU**
-2. Load & quantize the base model, add LoRA adapters
-3. Preprocess the TaiwanChat subset
-4. Fine‑tune with memory‑efficient settings
-5. Save locally under `./SmolLM2-135M-Instruct-TaiwanChat`
-6. Push the checkpoint to `huggingface.co/Luigi/SmolLM2-135M-Instruct-TaiwanChat`
-
----
-## Limitations
-- Fine‑tuned on a subset (~20 M tokens) for domain adaptation; may underperform on broader queries
-- No separate validation loop by default—monitor on a held‑out split if desired
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 4
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- num_epochs: 3
 
----
-## License
-- **Code**: Apache 2.0
-- **Data & weights**: CC BY‑NC 4.0 (non‑commercial)
-
----
-## Citation
-```bibtex
-@misc{SmolLM2TaiwanChat2025,
-  title        = {SmolLM2‑135M‑Instruct‑TaiwanChat},
-  author       = {Luigi Liu},
-  year         = {2025},
-  howpublished = {\url{https://huggingface.co/Luigi/SmolLM2-135M-Instruct-TaiwanChat}}
-}
-```
+### Framework versions
 
+- Transformers 4.51.3
+- Pytorch 2.6.0+xpu
+- Datasets 3.5.0
+- Tokenizers 0.21.1

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e6f75a45d501457abca6860e8a63fb18cd4396bd32e1a37de2fd319ad2565f66
+oid sha256:ff049bd3572ab5480dc285d0e177bf72fe0935b358167fa7c7595f4b428e687a
 size 538090408

runs/Apr22_10-34-08_luigi-inspiron135330/events.out.tfevents.1745289249.luigi-inspiron135330.505593.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99ec8bcbbe165f1aa0f67a5000b2b4da2fa861a59018add12473a03ebe48cca4
+size 5258

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad5ecda52d13dcd1e222159a20ec29dd5249c32517d3933c65710e27f3cb772d
-size 5368
+oid sha256:84050f9270a67639b4e89dc45333b7e10002df8a3d69d8957e2473f2089dba1c
+size 5304