Update README.md

README.md

@@ -1,3 +1,111 @@
 ---
-license: cc
+license: llama2
+library_name: peft
+tags:
+- typescript
+- instruction-tuning
+- code-generation
+- lora
+- peft
+base_model: codellama/CodeLlama-13b-hf
+model-index:
+- name: lora-out
+  results: []
+datasets:
+- mhhmm/typescript-instruct-20k
+language:
+- en
+metrics:
+- code_eval
+pipeline_tag: text-generation
 ---
+
+## Architecture
+
+![The Architecture](https://github.com/LeVuMinhHuy/brocode/blob/master/.pics/about-the-model.png?raw=true)
+
+## About
+
+This model is a fine-tuned version of [codellama/CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf).
+It achieves the following results on the evaluation set:
+- Loss: 0.4268
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 2
+- total_train_batch_size: 16
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 10
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 0.7555        | 0.01  | 1    | 0.7062          |
+| 0.7036        | 0.05  | 7    | 0.6673          |
+| 0.5422        | 0.1   | 14   | 0.5152          |
+| 0.5351        | 0.15  | 21   | 0.4866          |
+| 0.495         | 0.2   | 28   | 0.4688          |
+| 0.5651        | 0.25  | 35   | 0.4587          |
+| 0.5146        | 0.3   | 42   | 0.4486          |
+| 0.4955        | 0.35  | 49   | 0.4469          |
+| 0.5117        | 0.4   | 56   | 0.4432          |
+| 0.5245        | 0.45  | 63   | 0.4410          |
+| 0.5003        | 0.5   | 70   | 0.4371          |
+| 0.4502        | 0.55  | 77   | 0.4340          |
+| 0.527         | 0.6   | 84   | 0.4315          |
+| 0.48          | 0.65  | 91   | 0.4305          |
+| 0.448         | 0.7   | 98   | 0.4289          |
+| 0.5427        | 0.75  | 105  | 0.4289          |
+| 0.4715        | 0.8   | 112  | 0.4279          |
+| 0.5584        | 0.85  | 119  | 0.4276          |
+| 0.4936        | 0.9   | 126  | 0.4267          |
+| 0.4788        | 0.95  | 133  | 0.4268          |
+| 0.476         | 1.0   | 140  | 0.4268          |
+
+
+### Framework versions
+
+- Transformers 4.36.0.dev0
+- Pytorch 2.0.1+cu118
+- Datasets 2.15.0
+- Tokenizers 0.15.0
+- PEFT 0.6.0
+  
+### Evaluation
+
+I'm using MultiPL-E benchmark, the same as Code Llmama using in their paper
+
+
+| Modal                                   | Pass@k | Estimate | Num problems  |
+|-----------------------------------------|--------|----------|---------------|
+| Code LLama - Instruct 13B               | 1      | 39.0%    | 159           |
+| Our  13B                                | 1      | 42.4%    | 159           |
+
+How to reproduce my evaluation? Just run like the offical document of MultiPL-E: https://nuprl.github.io/MultiPL-E/tutorial.html, change the modal name by my model here: `mhhmm/typescript-instruct-20k-v2` 
+
+This is the code that I ran with Google Colab (using A100 40GB, yes, it requires that much GPU RAM)
+
+If you even have a stronger GPU, increase the --batch-size, or --completion-limit
+
+```
+!pip install --upgrade pip
+!pip install aiohttp numpy tqdm pytest datasets torch transformers sentencepiece
+!git clone https://github.com/nuprl/MultiPL-E
+%cd MultiPL-E
+!mkdir typescript
+!python3 automodel.py --name mhhmm/typescript-instruct-20k-v2 --root-dataset humaneval --lang ts --temperature 0.2 --batch-size 10 --completion-limit 20 --output-dir-prefix typescript
+%cd evaluation/src
+!python3 main.py --dir ../../typescript --output-dir ../../typescript --recursive
+!python3 pass_k.py ./typescript/*
+```
+