Model card updated after epoch 0

README.md

@@ -31,9 +31,9 @@ A 768M parameter Hierarchical Recurrent Memory (HRM) language model trained on h
 ### Configuration
 ```python
 Model Dimensions:
-  - d_model: 768
-  - n_heads: 12 (for compatibility, not used in Mamba)
-  - d_ff: 3072
+  - d_model: 1024
+  - n_heads: 16 (for compatibility, not used in Mamba)
+  - d_ff: 4096
   - H_layers: 12 (high-level hierarchy)
   - L_layers: 12 (low-level processing)
 
@@ -45,10 +45,10 @@ Mamba2 Settings:
   - ngroups: 1
 
 Training:
-  - Max halt steps: 8
+  - Max halt steps: 1
   - Block size: 1024
-  - Batch size: 32 (effective)
-  - Learning rate: 0.0002 → 1e-06
+  - Batch size: 64 (effective)
+  - Learning rate: 3e-05 → 1e-06
   - Weight decay: 0.1
 ```
 
@@ -58,16 +58,16 @@ Training:
 - **Tokenizer**: `t5-small` (T5 SentencePiece)
 - **Vocab Size**: 32100
 
-## Latest Performance (Epoch 1)
+## Latest Performance (Epoch 0)
 
-- **Validation Loss**: `8.3293`
-- **Validation Perplexity**: `4143.72`
+- **Validation Loss**: `10.3766`
+- **Validation Perplexity**: `32099.98`
 
 ## Usage
 
 ```python
 from transformers import T5Tokenizer
-from hrm_text1_modeling import HRMText1
+from hrm_text1_mamba1_donor import HRMText1
 
 tokenizer = T5Tokenizer.from_pretrained("t5-small")
 model = HRMText1.from_pretrained("Viharikvs/CMBA-768M-OpenWebMath")