|
|
--- |
|
|
language: |
|
|
- en |
|
|
- de |
|
|
- fr |
|
|
- it |
|
|
- pt |
|
|
- hi |
|
|
- es |
|
|
- th |
|
|
library_name: exllamav2 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- facebook |
|
|
- meta |
|
|
- pytorch |
|
|
- llama |
|
|
- llama-3 |
|
|
- safety |
|
|
license: llama3.2 |
|
|
base_model: |
|
|
- meta-llama/Llama-Guard-3-1B |
|
|
--- |
|
|
# Exl2 quants for [Llama-Guard-3-1B](https://huggingface.co/meta-llama/Llama-Guard-3-1B) |
|
|
|
|
|
This is a quantized model from the llama guard series for use with [exllamav2](https://github.com/turboderp/exllamav2).\ |
|
|
Please refer to the original model card for instructions on how to use this model. |
|
|
|
|
|
### Only use this version for testing, BF16 weights recommended for deployment or at least 8bpw. |
|
|
|
|
|
### BPW: |
|
|
|
|
|
[4.0](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/4.0bpw)\ |
|
|
[4.5](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/4.5bpw)\ |
|
|
[5.0](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/5.0bpw)\ |
|
|
[6.0](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/6.0bpw)\ |
|
|
[6.5](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/6.5bpw)\ |
|
|
[8.0](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/tree/8.0bpw)\ |
|
|
[measurement.json](https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2/blob/main/measurement.json) |
|
|
|
|
|
# How to download: |
|
|
|
|
|
### oobabooga's downloader |
|
|
|
|
|
use something like [download-model.py](https://github.com/oobabooga/text-generation-webui/blob/main/download-model.py) to download with python requests.\ |
|
|
Install requirements: |
|
|
|
|
|
```shell |
|
|
pip install requests tqdm |
|
|
``` |
|
|
|
|
|
Example for downloading 8bpw: |
|
|
|
|
|
```shell |
|
|
python download-model.py Anthonyg5005/Llama-Guard-3-1B-exl2:8.0bpw |
|
|
``` |
|
|
|
|
|
### huggingface-cli |
|
|
|
|
|
You may also use huggingface-cli\ |
|
|
To install it, install python hf-hub |
|
|
|
|
|
```shell |
|
|
pip install huggingface-hub |
|
|
``` |
|
|
|
|
|
Example for 8bpw: |
|
|
|
|
|
```shell |
|
|
huggingface-cli download Anthonyg5005/Llama-Guard-3-1B-exl2 --local-dir Llama-Guard-3-1B-exl2-exl2-8bpw --revision 8.0bpw |
|
|
``` |
|
|
### Git LFS (not recommended) |
|
|
|
|
|
I would recommend the http downloaders over using git, they can resume downloads if failed and are much easier to work with.\ |
|
|
Make sure to have git and git LFS installed.\ |
|
|
Example for 8bpw download with git: |
|
|
|
|
|
Have LFS file skip disabled |
|
|
```shell |
|
|
# windows |
|
|
set GIT_LFS_SKIP_SMUDGE=0 |
|
|
# linux |
|
|
export GIT_LFS_SKIP_SMUDGE=0 |
|
|
``` |
|
|
|
|
|
Clone repo branch |
|
|
```shell |
|
|
git clone https://huggingface.co/Anthonyg5005/Llama-Guard-3-1B-exl2 -b 8.0bpw |
|
|
``` |
