final model

log/debug_0.log

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d657e9c798fde57f1fef69fbedf64503cd82a9810aef7a412c729a79f44fe514
-size 22992153
+oid sha256:6a33bea23e477172105ae80f5cebc35cb70eef09e4d95bd808f4973dddf0930b
+size 22992728

runs/Nov06_21-16-12_leandro-16x-v100/events.out.tfevents.1636233372.leandro-16x-v100.4368.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dd2da3133b96068e4a12ef87687b875abc56e9140a4cd32b820111537cbdb872
-size 27534988
+oid sha256:2cd9716067e9f59fda4a7b67e25db6034c5b4465db63524decb1c80001219215
+size 27535087

wandb/run-20211106_211610-dtkf2u0m/files/output.log

@@ -37457,5 +37457,62 @@ To disable this warning, you can either:
 	- Avoid using `tokenizers` before the fork if possible
 	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
 huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+11/07/2021 18:23:37 - WARNING - huggingface_hub.repository - The progress bars may be unreliable.99616, 'steps': 149997, 'loss/train': 1.498703122138977}}}}}ils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+11/07/2021 18:23:37 - WARNING - huggingface_hub.repository - The progress bars may be unreliable.99616, 'steps': 149997, 'loss/train': 1.498703122138977}}}}}ils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:   0%|                                                                                                                     | 32.0k/48.8M [00:00<?, ?B/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  44%|███████████████████████████████████████████████▉                                                             | 21.4M/48.8M [00:02<00:02, 11.7MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb:  78%|█████████████████████████████████████████████████████████████████████████████████████▎                       | 38.2M/48.8M [00:03<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:18<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:18<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:18<00:00, 14.3MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:24<00:00, 2.12MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|██████████████████████████████████████████████████████��██████████████████████████████████████████████████████| 48.8M/48.8M [00:24<00:00, 2.12MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:24<00:00, 2.12MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:24<00:00, 2.12MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+Upload file wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 48.8M/48.8M [00:24<00:00, 2.12MB/s] behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
+To disable this warning, you can either:
+	- Avoid using `tokenizers` before the fork if possible
+	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
+huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
 To disable this warning, you can either:
 	- Avoid using `tokenizers` before the fork if possible

wandb/run-20211106_211610-dtkf2u0m/files/wandb-summary.json

@@ -1 +1 @@
-{"lr": 5.63230573291662e-14, "samples": 28800000, "steps": 149999, "loss/train": 1.469771146774292, "_runtime": 76030, "_timestamp": 1636309400, "_step": 150009, "loss/eval": 1.2280396223068237, "perplexity": 3.414529323577881}
+{"lr": 5.63230573291662e-14, "samples": 28800000, "steps": 149999, "loss/train": 1.469771146774292, "_runtime": 76270, "_timestamp": 1636309640, "_step": 150010, "loss/eval": 1.2280396223068237, "perplexity": 3.414529323577881}

wandb/run-20211106_211610-dtkf2u0m/logs/debug-internal.log

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fc35461595e40ec481bdea871164294be3bb20552c28c1e92d57834c8091c201
-size 67412844
+oid sha256:53d06c5a631b6282f504e2581be60edaf25b6922e2ee3189c9140c2e3460d2c4
+size 67421355

wandb/run-20211106_211610-dtkf2u0m/run-dtkf2u0m.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b19419e59b05006f6381c513ec0f2b32c80e38dfbfb1571aa3672553faec0dbe
-size 51160299
+oid sha256:65fd3ed9e21bd18b6b28e3984baf05f344d8d2728f5429650ae99a1c9aae34c8
+size 51195135