Loading pretrained model
Loading datasets
Training
Trainable parameters: 0.145% (10.486M/7248.024M)
Starting training..., iters: 600

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:32,  1.36s/it]
Calculating loss...:   8%|▊         | 2/25 [00:02<00:33,  1.47s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:04<00:34,  1.55s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:05<00:30,  1.47s/it]
Calculating loss...:  20%|██        | 5/25 [00:07<00:28,  1.43s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:09<00:34,  1.82s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:11<00:31,  1.72s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:13<00:29,  1.74s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:14<00:27,  1.69s/it]
Calculating loss...:  40%|████      | 10/25 [00:16<00:25,  1.68s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:17<00:21,  1.56s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:19<00:20,  1.56s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:20<00:17,  1.50s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:22<00:16,  1.48s/it]
Calculating loss...:  60%|██████    | 15/25 [00:23<00:15,  1.60s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:25<00:14,  1.58s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:26<00:12,  1.56s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:28<00:11,  1.63s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:30<00:10,  1.68s/it]
Calculating loss...:  80%|████████  | 20/25 [00:31<00:07,  1.59s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:33<00:06,  1.52s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:34<00:04,  1.44s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:36<00:02,  1.45s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:37<00:01,  1.51s/it]
Calculating loss...: 100%|██████████| 25/25 [00:39<00:00,  1.60s/it]
Calculating loss...: 100%|██████████| 25/25 [00:39<00:00,  1.58s/it]
Iter 1: Val loss 2.214, Val took 39.464s
Iter 10: Train loss 1.065, Learning Rate 1.000e-05, It/sec 0.362, Tokens/sec 431.356, Trained Tokens 11910, Peak mem 24.920 GB
Iter 20: Train loss 0.755, Learning Rate 1.000e-05, It/sec 0.263, Tokens/sec 350.838, Trained Tokens 25233, Peak mem 37.081 GB
Iter 30: Train loss 0.688, Learning Rate 1.000e-05, It/sec 0.352, Tokens/sec 429.868, Trained Tokens 37431, Peak mem 37.081 GB
Iter 40: Train loss 0.557, Learning Rate 1.000e-05, It/sec 0.340, Tokens/sec 403.603, Trained Tokens 49302, Peak mem 37.081 GB
Iter 50: Train loss 0.663, Learning Rate 1.000e-05, It/sec 0.296, Tokens/sec 387.318, Trained Tokens 62366, Peak mem 37.081 GB
Iter 60: Train loss 0.609, Learning Rate 1.000e-05, It/sec 0.345, Tokens/sec 424.207, Trained Tokens 74645, Peak mem 37.081 GB
Iter 70: Train loss 0.614, Learning Rate 1.000e-05, It/sec 0.344, Tokens/sec 421.161, Trained Tokens 86901, Peak mem 37.081 GB
Iter 80: Train loss 0.607, Learning Rate 1.000e-05, It/sec 0.348, Tokens/sec 423.462, Trained Tokens 99079, Peak mem 37.081 GB
Iter 90: Train loss 0.556, Learning Rate 1.000e-05, It/sec 0.373, Tokens/sec 439.181, Trained Tokens 110838, Peak mem 37.081 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:43,  1.83s/it]
Calculating loss...:   8%|▊         | 2/25 [00:03<00:37,  1.65s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:04<00:34,  1.55s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:06<00:31,  1.51s/it]
Calculating loss...:  20%|██        | 5/25 [00:07<00:29,  1.46s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:09<00:28,  1.48s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:10<00:26,  1.49s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:11<00:24,  1.44s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:13<00:22,  1.41s/it]
Calculating loss...:  40%|████      | 10/25 [00:14<00:21,  1.45s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:16<00:21,  1.54s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:17<00:19,  1.49s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:19<00:18,  1.50s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:21<00:16,  1.50s/it]
Calculating loss...:  60%|██████    | 15/25 [00:22<00:15,  1.51s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:24<00:15,  1.72s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:26<00:13,  1.67s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:29<00:13,  1.99s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:30<00:10,  1.82s/it]
Calculating loss...:  80%|████████  | 20/25 [00:33<00:10,  2.12s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:34<00:07,  1.89s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:36<00:05,  1.75s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:37<00:03,  1.68s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:38<00:01,  1.61s/it]
Calculating loss...: 100%|██████████| 25/25 [00:40<00:00,  1.60s/it]
Calculating loss...: 100%|██████████| 25/25 [00:40<00:00,  1.62s/it]
Iter 100: Val loss 0.553, Val took 40.597s
Iter 100: Train loss 0.487, Learning Rate 1.000e-05, It/sec 0.317, Tokens/sec 358.772, Trained Tokens 122142, Peak mem 37.081 GB
Iter 100: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000100_adapters.safetensors.
Iter 110: Train loss 0.556, Learning Rate 1.000e-05, It/sec 0.366, Tokens/sec 430.562, Trained Tokens 133896, Peak mem 37.081 GB
Iter 120: Train loss 0.550, Learning Rate 1.000e-05, It/sec 0.348, Tokens/sec 437.090, Trained Tokens 146466, Peak mem 37.081 GB
Iter 130: Train loss 0.499, Learning Rate 1.000e-05, It/sec 0.349, Tokens/sec 430.415, Trained Tokens 158812, Peak mem 37.081 GB
Iter 140: Train loss 0.393, Learning Rate 1.000e-05, It/sec 0.436, Tokens/sec 465.004, Trained Tokens 169489, Peak mem 37.081 GB
Iter 150: Train loss 0.506, Learning Rate 1.000e-05, It/sec 0.372, Tokens/sec 438.469, Trained Tokens 181272, Peak mem 37.081 GB
Iter 160: Train loss 0.484, Learning Rate 1.000e-05, It/sec 0.375, Tokens/sec 443.286, Trained Tokens 193104, Peak mem 37.081 GB
Iter 170: Train loss 0.480, Learning Rate 1.000e-05, It/sec 0.399, Tokens/sec 448.274, Trained Tokens 204339, Peak mem 37.081 GB
Iter 180: Train loss 0.491, Learning Rate 1.000e-05, It/sec 0.374, Tokens/sec 442.924, Trained Tokens 216181, Peak mem 37.081 GB
Iter 190: Train loss 0.612, Learning Rate 1.000e-05, It/sec 0.259, Tokens/sec 348.364, Trained Tokens 229607, Peak mem 37.123 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:03<01:30,  3.75s/it]
Calculating loss...:   8%|▊         | 2/25 [00:05<01:00,  2.64s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:07<00:54,  2.47s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:10<00:49,  2.36s/it]
Calculating loss...:  20%|██        | 5/25 [00:11<00:39,  1.99s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:13<00:37,  1.96s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:14<00:31,  1.74s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:16<00:28,  1.68s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:19<00:33,  2.10s/it]
Calculating loss...:  40%|████      | 10/25 [00:21<00:30,  2.02s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:22<00:26,  1.88s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:24<00:23,  1.83s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:25<00:20,  1.72s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:27<00:19,  1.75s/it]
Calculating loss...:  60%|██████    | 15/25 [00:30<00:21,  2.15s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:32<00:17,  1.99s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:34<00:15,  1.95s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:35<00:12,  1.75s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:37<00:10,  1.72s/it]
Calculating loss...:  80%|████████  | 20/25 [00:39<00:08,  1.78s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:40<00:06,  1.65s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:41<00:04,  1.64s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:43<00:03,  1.70s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:45<00:01,  1.63s/it]
Calculating loss...: 100%|██████████| 25/25 [00:46<00:00,  1.59s/it]
Calculating loss...: 100%|██████████| 25/25 [00:46<00:00,  1.87s/it]
Iter 200: Val loss 0.499, Val took 46.784s
Iter 200: Train loss 0.426, Learning Rate 1.000e-05, It/sec 0.364, Tokens/sec 420.021, Trained Tokens 241139, Peak mem 37.123 GB
Iter 200: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000200_adapters.safetensors.
Iter 210: Train loss 0.402, Learning Rate 1.000e-05, It/sec 0.364, Tokens/sec 403.442, Trained Tokens 252219, Peak mem 37.123 GB
Iter 220: Train loss 0.432, Learning Rate 1.000e-05, It/sec 0.408, Tokens/sec 467.043, Trained Tokens 263665, Peak mem 37.123 GB
Iter 230: Train loss 0.562, Learning Rate 1.000e-05, It/sec 0.369, Tokens/sec 447.334, Trained Tokens 275796, Peak mem 37.123 GB
Iter 240: Train loss 0.491, Learning Rate 1.000e-05, It/sec 0.396, Tokens/sec 461.514, Trained Tokens 287447, Peak mem 37.123 GB
Iter 250: Train loss 0.460, Learning Rate 1.000e-05, It/sec 0.391, Tokens/sec 453.744, Trained Tokens 299037, Peak mem 37.123 GB
Iter 260: Train loss 0.462, Learning Rate 1.000e-05, It/sec 0.403, Tokens/sec 470.594, Trained Tokens 310708, Peak mem 37.123 GB
Iter 270: Train loss 0.411, Learning Rate 1.000e-05, It/sec 0.397, Tokens/sec 455.131, Trained Tokens 322185, Peak mem 37.123 GB
Iter 280: Train loss 0.388, Learning Rate 1.000e-05, It/sec 0.386, Tokens/sec 447.749, Trained Tokens 333784, Peak mem 37.123 GB
Iter 290: Train loss 0.506, Learning Rate 1.000e-05, It/sec 0.401, Tokens/sec 459.746, Trained Tokens 345244, Peak mem 37.123 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:33,  1.40s/it]
Calculating loss...:   8%|▊         | 2/25 [00:04<00:48,  2.13s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:05<00:40,  1.83s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:07<00:35,  1.70s/it]
Calculating loss...:  20%|██        | 5/25 [00:08<00:32,  1.62s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:09<00:29,  1.55s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:11<00:26,  1.48s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:12<00:24,  1.46s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:14<00:23,  1.49s/it]
Calculating loss...:  40%|████      | 10/25 [00:15<00:22,  1.51s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:17<00:21,  1.50s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:18<00:19,  1.52s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:20<00:19,  1.60s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:21<00:16,  1.52s/it]
Calculating loss...:  60%|██████    | 15/25 [00:24<00:18,  1.88s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:25<00:15,  1.70s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:27<00:12,  1.60s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:29<00:11,  1.66s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:30<00:10,  1.69s/it]
Calculating loss...:  80%|████████  | 20/25 [00:32<00:08,  1.66s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:33<00:06,  1.54s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:35<00:04,  1.57s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:37<00:03,  1.73s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:38<00:01,  1.63s/it]
Calculating loss...: 100%|██████████| 25/25 [00:40<00:00,  1.54s/it]
Calculating loss...: 100%|██████████| 25/25 [00:40<00:00,  1.61s/it]
Iter 300: Val loss 0.475, Val took 40.184s
Iter 300: Train loss 0.520, Learning Rate 1.000e-05, It/sec 0.420, Tokens/sec 467.207, Trained Tokens 356356, Peak mem 37.123 GB
Iter 300: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000300_adapters.safetensors.
Iter 310: Train loss 0.429, Learning Rate 1.000e-05, It/sec 0.430, Tokens/sec 478.254, Trained Tokens 367482, Peak mem 37.123 GB
Iter 320: Train loss 0.420, Learning Rate 1.000e-05, It/sec 0.373, Tokens/sec 439.972, Trained Tokens 379266, Peak mem 37.123 GB
Iter 330: Train loss 0.480, Learning Rate 1.000e-05, It/sec 0.399, Tokens/sec 462.754, Trained Tokens 390875, Peak mem 37.123 GB
Iter 340: Train loss 0.456, Learning Rate 1.000e-05, It/sec 0.404, Tokens/sec 456.805, Trained Tokens 402182, Peak mem 37.123 GB
Iter 350: Train loss 0.639, Learning Rate 1.000e-05, It/sec 0.300, Tokens/sec 410.751, Trained Tokens 415852, Peak mem 37.972 GB
Iter 360: Train loss 0.522, Learning Rate 1.000e-05, It/sec 0.407, Tokens/sec 474.435, Trained Tokens 427505, Peak mem 37.972 GB
Iter 370: Train loss 0.525, Learning Rate 1.000e-05, It/sec 0.332, Tokens/sec 434.123, Trained Tokens 440587, Peak mem 37.972 GB
Iter 380: Train loss 0.424, Learning Rate 1.000e-05, It/sec 0.374, Tokens/sec 446.958, Trained Tokens 452536, Peak mem 37.972 GB
Iter 390: Train loss 0.428, Learning Rate 1.000e-05, It/sec 0.357, Tokens/sec 444.952, Trained Tokens 464991, Peak mem 37.972 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:37,  1.55s/it]
Calculating loss...:   8%|▊         | 2/25 [00:03<00:34,  1.52s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:04<00:36,  1.64s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:08<00:48,  2.29s/it]
Calculating loss...:  20%|██        | 5/25 [00:09<00:39,  1.96s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:10<00:33,  1.77s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:12<00:30,  1.70s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:13<00:27,  1.63s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:16<00:29,  1.83s/it]
Calculating loss...:  40%|████      | 10/25 [00:17<00:25,  1.68s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:18<00:21,  1.56s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:20<00:19,  1.54s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:21<00:17,  1.48s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:23<00:15,  1.44s/it]
Calculating loss...:  60%|██████    | 15/25 [00:27<00:24,  2.41s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:29<00:20,  2.27s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:30<00:15,  2.00s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:33<00:15,  2.19s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:35<00:12,  2.07s/it]
Calculating loss...:  80%|████████  | 20/25 [00:36<00:09,  1.85s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:38<00:06,  1.68s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:39<00:04,  1.55s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:40<00:03,  1.50s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:42<00:01,  1.47s/it]
Calculating loss...: 100%|██████████| 25/25 [00:43<00:00,  1.51s/it]
Calculating loss...: 100%|██████████| 25/25 [00:43<00:00,  1.75s/it]
Iter 400: Val loss 0.508, Val took 43.651s
Iter 400: Train loss 0.352, Learning Rate 1.000e-05, It/sec 0.346, Tokens/sec 403.495, Trained Tokens 476660, Peak mem 37.972 GB
Iter 400: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000400_adapters.safetensors.
Iter 410: Train loss 0.362, Learning Rate 1.000e-05, It/sec 0.354, Tokens/sec 436.195, Trained Tokens 488988, Peak mem 37.972 GB
Iter 420: Train loss 0.319, Learning Rate 1.000e-05, It/sec 0.412, Tokens/sec 477.171, Trained Tokens 500576, Peak mem 37.972 GB
Iter 430: Train loss 0.347, Learning Rate 1.000e-05, It/sec 0.394, Tokens/sec 461.272, Trained Tokens 512284, Peak mem 37.972 GB
Iter 440: Train loss 0.370, Learning Rate 1.000e-05, It/sec 0.375, Tokens/sec 436.054, Trained Tokens 523900, Peak mem 37.972 GB
Iter 450: Train loss 0.370, Learning Rate 1.000e-05, It/sec 0.375, Tokens/sec 460.385, Trained Tokens 536188, Peak mem 37.972 GB
Iter 460: Train loss 0.362, Learning Rate 1.000e-05, It/sec 0.401, Tokens/sec 460.743, Trained Tokens 547689, Peak mem 37.972 GB
Iter 470: Train loss 0.343, Learning Rate 1.000e-05, It/sec 0.391, Tokens/sec 464.058, Trained Tokens 559569, Peak mem 37.972 GB
Iter 480: Train loss 0.375, Learning Rate 1.000e-05, It/sec 0.358, Tokens/sec 445.015, Trained Tokens 572002, Peak mem 37.972 GB
Iter 490: Train loss 0.338, Learning Rate 1.000e-05, It/sec 0.408, Tokens/sec 471.137, Trained Tokens 583561, Peak mem 37.972 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:35,  1.49s/it]
Calculating loss...:   8%|▊         | 2/25 [00:02<00:32,  1.40s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:04<00:31,  1.41s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:05<00:30,  1.44s/it]
Calculating loss...:  20%|██        | 5/25 [00:07<00:31,  1.57s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:08<00:27,  1.47s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:10<00:28,  1.57s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:11<00:25,  1.49s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:13<00:24,  1.54s/it]
Calculating loss...:  40%|████      | 10/25 [00:14<00:22,  1.50s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:16<00:20,  1.47s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:17<00:19,  1.48s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:19<00:17,  1.48s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:20<00:16,  1.48s/it]
Calculating loss...:  60%|██████    | 15/25 [00:23<00:18,  1.83s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:24<00:15,  1.73s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:26<00:12,  1.61s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:27<00:10,  1.53s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:28<00:08,  1.47s/it]
Calculating loss...:  80%|████████  | 20/25 [00:30<00:07,  1.41s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:31<00:05,  1.45s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:33<00:04,  1.44s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:34<00:02,  1.43s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:36<00:01,  1.47s/it]
Calculating loss...: 100%|██████████| 25/25 [00:37<00:00,  1.47s/it]
Calculating loss...: 100%|██████████| 25/25 [00:37<00:00,  1.51s/it]
Iter 500: Val loss 0.472, Val took 37.645s
Iter 500: Train loss 0.357, Learning Rate 1.000e-05, It/sec 0.371, Tokens/sec 445.737, Trained Tokens 595562, Peak mem 37.972 GB
Iter 500: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000500_adapters.safetensors.
Iter 510: Train loss 0.461, Learning Rate 1.000e-05, It/sec 0.360, Tokens/sec 451.720, Trained Tokens 608094, Peak mem 37.972 GB
Iter 520: Train loss 0.350, Learning Rate 1.000e-05, It/sec 0.396, Tokens/sec 460.678, Trained Tokens 619734, Peak mem 37.972 GB
Iter 530: Train loss 0.403, Learning Rate 1.000e-05, It/sec 0.380, Tokens/sec 459.285, Trained Tokens 631829, Peak mem 37.972 GB
Iter 540: Train loss 0.311, Learning Rate 1.000e-05, It/sec 0.436, Tokens/sec 473.710, Trained Tokens 642682, Peak mem 37.972 GB
Iter 550: Train loss 0.350, Learning Rate 1.000e-05, It/sec 0.393, Tokens/sec 459.128, Trained Tokens 654367, Peak mem 37.972 GB
Iter 560: Train loss 0.484, Learning Rate 1.000e-05, It/sec 0.319, Tokens/sec 420.622, Trained Tokens 667572, Peak mem 37.972 GB
Iter 570: Train loss 0.390, Learning Rate 1.000e-05, It/sec 0.394, Tokens/sec 464.738, Trained Tokens 679358, Peak mem 37.972 GB
Iter 580: Train loss 0.364, Learning Rate 1.000e-05, It/sec 0.399, Tokens/sec 460.012, Trained Tokens 690891, Peak mem 37.972 GB
Iter 590: Train loss 0.337, Learning Rate 1.000e-05, It/sec 0.401, Tokens/sec 468.869, Trained Tokens 702595, Peak mem 37.972 GB

Calculating loss...:   0%|          | 0/25 [00:00<?, ?it/s]
Calculating loss...:   4%|▍         | 1/25 [00:01<00:42,  1.78s/it]
Calculating loss...:   8%|▊         | 2/25 [00:03<00:35,  1.56s/it]
Calculating loss...:  12%|█▏        | 3/25 [00:04<00:32,  1.46s/it]
Calculating loss...:  16%|█▌        | 4/25 [00:06<00:30,  1.47s/it]
Calculating loss...:  20%|██        | 5/25 [00:07<00:30,  1.51s/it]
Calculating loss...:  24%|██▍       | 6/25 [00:09<00:28,  1.50s/it]
Calculating loss...:  28%|██▊       | 7/25 [00:10<00:26,  1.45s/it]
Calculating loss...:  32%|███▏      | 8/25 [00:11<00:24,  1.46s/it]
Calculating loss...:  36%|███▌      | 9/25 [00:13<00:24,  1.56s/it]
Calculating loss...:  40%|████      | 10/25 [00:15<00:24,  1.61s/it]
Calculating loss...:  44%|████▍     | 11/25 [00:17<00:25,  1.79s/it]
Calculating loss...:  48%|████▊     | 12/25 [00:19<00:22,  1.70s/it]
Calculating loss...:  52%|█████▏    | 13/25 [00:20<00:19,  1.61s/it]
Calculating loss...:  56%|█████▌    | 14/25 [00:21<00:16,  1.53s/it]
Calculating loss...:  60%|██████    | 15/25 [00:23<00:14,  1.47s/it]
Calculating loss...:  64%|██████▍   | 16/25 [00:24<00:12,  1.40s/it]
Calculating loss...:  68%|██████▊   | 17/25 [00:25<00:10,  1.36s/it]
Calculating loss...:  72%|███████▏  | 18/25 [00:27<00:09,  1.41s/it]
Calculating loss...:  76%|███████▌  | 19/25 [00:28<00:08,  1.37s/it]
Calculating loss...:  80%|████████  | 20/25 [00:29<00:06,  1.36s/it]
Calculating loss...:  84%|████████▍ | 21/25 [00:31<00:05,  1.40s/it]
Calculating loss...:  88%|████████▊ | 22/25 [00:32<00:04,  1.36s/it]
Calculating loss...:  92%|█████████▏| 23/25 [00:34<00:02,  1.42s/it]
Calculating loss...:  96%|█████████▌| 24/25 [00:35<00:01,  1.44s/it]
Calculating loss...: 100%|██████████| 25/25 [00:36<00:00,  1.41s/it]
Calculating loss...: 100%|██████████| 25/25 [00:36<00:00,  1.48s/it]
Iter 600: Val loss 0.427, Val took 36.973s
Iter 600: Train loss 0.346, Learning Rate 1.000e-05, It/sec 0.427, Tokens/sec 474.136, Trained Tokens 713698, Peak mem 37.972 GB
Iter 600: Saved adapter weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors and models/lora/mistral_lora_telegram_20251111_114741/0000600_adapters.safetensors.
Saved final weights to models/lora/mistral_lora_telegram_20251111_114741/adapters.safetensors.
Testing

Calculating loss...:   0%|          | 0/50 [00:00<?, ?it/s]
Calculating loss...:   2%|▏         | 1/50 [00:01<01:12,  1.49s/it]
Calculating loss...:   4%|▍         | 2/50 [00:02<01:04,  1.35s/it]
Calculating loss...:   6%|▌         | 3/50 [00:04<01:11,  1.52s/it]
Calculating loss...:   8%|▊         | 4/50 [00:06<01:10,  1.54s/it]
Calculating loss...:  10%|█         | 5/50 [00:07<01:08,  1.52s/it]
Calculating loss...:  12%|█▏        | 6/50 [00:09<01:07,  1.54s/it]
Calculating loss...:  14%|█▍        | 7/50 [00:10<01:05,  1.52s/it]
Calculating loss...:  16%|█▌        | 8/50 [00:11<01:01,  1.46s/it]
Calculating loss...:  18%|█▊        | 9/50 [00:13<00:57,  1.40s/it]
Calculating loss...:  20%|██        | 10/50 [00:14<00:59,  1.49s/it]
Calculating loss...:  22%|██▏       | 11/50 [00:16<00:57,  1.47s/it]
Calculating loss...:  24%|██▍       | 12/50 [00:17<00:53,  1.41s/it]
Calculating loss...:  26%|██▌       | 13/50 [00:18<00:51,  1.38s/it]
Calculating loss...:  28%|██▊       | 14/50 [00:20<00:49,  1.39s/it]
Calculating loss...:  30%|███       | 15/50 [00:21<00:47,  1.35s/it]
Calculating loss...:  32%|███▏      | 16/50 [00:23<00:48,  1.44s/it]
Calculating loss...:  34%|███▍      | 17/50 [00:24<00:49,  1.50s/it]
Calculating loss...:  36%|███▌      | 18/50 [00:26<00:48,  1.50s/it]
Calculating loss...:  38%|███▊      | 19/50 [00:27<00:45,  1.45s/it]
Calculating loss...:  40%|████      | 20/50 [00:29<00:42,  1.42s/it]
Calculating loss...:  42%|████▏     | 21/50 [00:30<00:43,  1.49s/it]
Calculating loss...:  44%|████▍     | 22/50 [00:32<00:41,  1.49s/it]
Calculating loss...:  46%|████▌     | 23/50 [00:33<00:38,  1.44s/it]
Calculating loss...:  48%|████▊     | 24/50 [00:34<00:36,  1.41s/it]
Calculating loss...:  50%|█████     | 25/50 [00:36<00:37,  1.48s/it]
Calculating loss...:  52%|█████▏    | 26/50 [00:37<00:33,  1.41s/it]
Calculating loss...:  54%|█████▍    | 27/50 [00:39<00:31,  1.39s/it]
Calculating loss...:  56%|█████▌    | 28/50 [00:43<00:52,  2.39s/it]
Calculating loss...:  58%|█████▊    | 29/50 [00:46<00:50,  2.39s/it]
Calculating loss...:  60%|██████    | 30/50 [00:47<00:43,  2.17s/it]
Calculating loss...:  62%|██████▏   | 31/50 [00:49<00:36,  1.94s/it]
Calculating loss...:  64%|██████▍   | 32/50 [00:51<00:38,  2.16s/it]
Calculating loss...:  66%|██████▌   | 33/50 [00:53<00:34,  2.04s/it]
Calculating loss...:  68%|██████▊   | 34/50 [00:55<00:30,  1.88s/it]
Calculating loss...:  70%|███████   | 35/50 [00:57<00:31,  2.08s/it]
Calculating loss...:  72%|███████▏  | 36/50 [00:59<00:26,  1.88s/it]
Calculating loss...:  74%|███████▍  | 37/50 [01:00<00:24,  1.85s/it]
Calculating loss...:  76%|███████▌  | 38/50 [01:02<00:21,  1.79s/it]
Calculating loss...:  78%|███████▊  | 39/50 [01:03<00:18,  1.68s/it]
Calculating loss...:  80%|████████  | 40/50 [01:06<00:19,  1.99s/it]
Calculating loss...:  82%|████████▏ | 41/50 [01:08<00:16,  1.84s/it]
Calculating loss...:  84%|████████▍ | 42/50 [01:09<00:14,  1.82s/it]
Calculating loss...:  86%|████████▌ | 43/50 [01:11<00:12,  1.77s/it]
Calculating loss...:  88%|████████▊ | 44/50 [01:13<00:10,  1.77s/it]
Calculating loss...:  90%|█████████ | 45/50 [01:15<00:08,  1.78s/it]
Calculating loss...:  92%|█████████▏| 46/50 [01:16<00:06,  1.70s/it]
Calculating loss...:  94%|█████████▍| 47/50 [01:18<00:04,  1.63s/it]
Calculating loss...:  96%|█████████▌| 48/50 [01:19<00:03,  1.56s/it]
Calculating loss...:  98%|█████████▊| 49/50 [01:20<00:01,  1.51s/it]
Calculating loss...: 100%|██████████| 50/50 [01:22<00:00,  1.59s/it]
Calculating loss...: 100%|██████████| 50/50 [01:22<00:00,  1.65s/it]
Test loss 0.474, Test ppl 1.606.