Defend the honour of the HF tokenizer
Browse files
README.md
CHANGED
|
@@ -16,12 +16,7 @@ widget:
|
|
| 16 |
---
|
| 17 |
# Model Card for Mixtral-8x7B
|
| 18 |
|
| 19 |
-
###
|
| 20 |
-
|
| 21 |
-
> [!CAUTION]
|
| 22 |
-
> ⚠️
|
| 23 |
-
> The `transformers` tokenizer might give incorrect results as it has not been tested by the Mistral team. To make sure that your encoding and decoding is correct, please use mistral-common as shown below:
|
| 24 |
-
|
| 25 |
|
| 26 |
```py
|
| 27 |
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
|
|
|
|
| 16 |
---
|
| 17 |
# Model Card for Mixtral-8x7B
|
| 18 |
|
| 19 |
+
### Tokenization with `mistral-common`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
```py
|
| 22 |
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
|