Update README.md
Browse files
README.md
CHANGED
|
@@ -15,11 +15,11 @@ tags:
|
|
| 15 |
Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1)
|
| 16 |
1. Do not forget about `<|User|>` and `<|Assistant|>` tokens! - Or use a chat template formatter
|
| 17 |
2. Obtain the latest `llama.cpp` at https://github.com/ggerganov/llama.cpp
|
| 18 |
-
3. Example with
|
| 19 |
```bash
|
| 20 |
./llama.cpp/llama-cli \
|
| 21 |
-
--model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-
|
| 22 |
-
--cache-type-k
|
| 23 |
--threads 16 \
|
| 24 |
--prompt '<|User|>What is 1+1?<|Assistant|>' \
|
| 25 |
-no-cnv
|
|
@@ -38,12 +38,12 @@ Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](h
|
|
| 38 |
4. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers.
|
| 39 |
```bash
|
| 40 |
./llama.cpp/llama-cli \
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
```
|
| 48 |
|
| 49 |
# Finetune LLMs 2-5x faster with 70% less memory via Unsloth!
|
|
|
|
| 15 |
Or you can view more detailed instructions here: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1)
|
| 16 |
1. Do not forget about `<|User|>` and `<|Assistant|>` tokens! - Or use a chat template formatter
|
| 17 |
2. Obtain the latest `llama.cpp` at https://github.com/ggerganov/llama.cpp
|
| 18 |
+
3. Example with Q5_0 K quantized cache **Notice -no-cnv disables auto conversation mode**
|
| 19 |
```bash
|
| 20 |
./llama.cpp/llama-cli \
|
| 21 |
+
--model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q2_K_XS.gguf \
|
| 22 |
+
--cache-type-k q5_0 \
|
| 23 |
--threads 16 \
|
| 24 |
--prompt '<|User|>What is 1+1?<|Assistant|>' \
|
| 25 |
-no-cnv
|
|
|
|
| 38 |
4. If you have a GPU (RTX 4090 for example) with 24GB, you can offload multiple layers to the GPU for faster processing. If you have multiple GPUs, you can probably offload more layers.
|
| 39 |
```bash
|
| 40 |
./llama.cpp/llama-cli \
|
| 41 |
+
--model unsloth/DeepSeek-R1-GGUF/DeepSeek-R1-Q2_K_XS.gguf \
|
| 42 |
+
--cache-type-k q5_0 \
|
| 43 |
+
--threads 16
|
| 44 |
+
--prompt '<|User|>What is 1+1?<|Assistant|>'
|
| 45 |
+
--n-gpu-layers 20 \
|
| 46 |
+
-no-cnv
|
| 47 |
```
|
| 48 |
|
| 49 |
# Finetune LLMs 2-5x faster with 70% less memory via Unsloth!
|