Update deepspeed config

README.md

@@ -31,16 +31,48 @@ We present the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks.
 --------
 #### Notes.
  - <sup>1</sup> Following RoBERTa, for RTE, MRPC, STS-B, we fine-tune the tasks based on [DeBERTa-Large-MNLI](https://huggingface.co/microsoft/deberta-large-mnli), [DeBERTa-XLarge-MNLI](https://huggingface.co/microsoft/deberta-xlarge-mnli), [DeBERTa-V2-XLarge-MNLI](https://huggingface.co/microsoft/deberta-v2-xlarge-mnli), [DeBERTa-V2-XXLarge-MNLI](https://huggingface.co/microsoft/deberta-v2-xxlarge-mnli). The results of SST-2/QQP/QNLI/SQuADv2 will also be slightly improved when start from MNLI fine-tuned models, however, we only report the numbers fine-tuned from pretrained base models for those 4 tasks.
- - <sup>2</sup> To try the **XXLarge** model with **[HF transformers](https://huggingface.co/transformers/main_classes/trainer.html)**, you need to specify **--sharded_ddp**
+ - <sup>2</sup> To try the **XXLarge** model with **[HF transformers](https://huggingface.co/transformers/main_classes/trainer.html)**, we recommand using **deepspeed** as it's faster and saves memory.
  
+Run with `Deepspeed`,
+
+```bash
+pip install datasets
+pip install deepspeed
+
+# Download the deepspeed config file
+wget https://huggingface.co/microsoft/deberta-v2-xxlarge/resolve/main/ds_config.json -O ds_config.json
+
+export TASK_NAME=mnli
+output_dir="ds_results"
+num_gpus=8
+batch_size=8
+python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
+  run_glue.py \
+  --model_name_or_path microsoft/deberta-v2-xxlarge \
+  --task_name $TASK_NAME \
+  --do_train \
+  --do_eval \
+  --max_seq_length 256 \
+  --per_device_train_batch_size ${batch_size} \
+  --learning_rate 3e-6 \
+  --num_train_epochs 3 \
+  --output_dir $output_dir \
+  --overwrite_output_dir \
+  --logging_steps 10 \
+  --logging_dir $output_dir \
+  --deepspeed ds_config.json
+```
+
+You can also run with `--sharded_ddp`
 ```bash  
 cd transformers/examples/text-classification/
-export TASK_NAME=mrpc
+export TASK_NAME=mnli
 python -m torch.distributed.launch --nproc_per_node=8 run_glue.py   --model_name_or_path microsoft/deberta-v2-xxlarge   \
---task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 4   \
+--task_name $TASK_NAME   --do_train   --do_eval   --max_seq_length 256   --per_device_train_batch_size 8   \
 --learning_rate 3e-6   --num_train_epochs 3   --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --sharded_ddp --fp16
 ```
 
+
 ### Citation
 
 If you find DeBERTa useful for your work, please cite the following paper:

ds_config.json

@@ -0,0 +1,23 @@
+{
+  "fp16": {
+
+    "enabled": true,
+
+    "initial_scale_power": 12
+
+  },
+  "zero_optimization": {
+
+    "stage": 2,
+
+    "reduce_bucket_size": 5e7,
+
+    "allgather_bucket_size": 1.25e9,
+
+    "overlap_comm": true,
+
+    "contiguous_gradients": true
+
+  },
+  "zero_allow_untested_optimizer": true
+}