Update README.md

README.md

@@ -1,86 +1,101 @@
 ---
+license: apache-2.0
 language:
 - en
 - es
 - it
 - de
 - fr
-license: apache-2.0
+tags:
+- moe
 ---
 
 # Model Card for Mixtral-8x22B-Instruct-v0.1
 
+The Mixtral-8x22B-Instruct-v0.1 Large Language Model (LLM) is an instruct fine-tuned version of the Mixtral-8x22B-v0.1.
+
+Mixtral-8x22B-v0.1 has the following characteristics:
+- 140.6B parameters
+- 39.1B active parameters
+- 64k context window
+- 32768 vocab size
+- Supports function calling
+
+## How to use
+
+It is recommended to use `mistralai/Mixtral-8x22B-Instruct-v0.1` with [mistral_inference](https://github.com/mistralai/mistral-inference) and [mistral_common](https://github.com/mistralai/mistral-common). For HF `transformers` code snippets, please keep scrolling.
+
+## Generate with `mistral_inference` and `mistral_common`
+
+### Install dependencies
+```
+pip install mistral_inference mistral_common
+```
+
+### Download model
+
+```py
+from huggingface_hub import snapshot_download
+from pathlib import Path
+
+mistral_models_path = Path.home().joinpath('mistral_models', '8x22B-Instruct-v0.1')
+mistral_models_path.mkdir(parents=True, exist_ok=True)
+
+snapshot_download(repo_id="mistralai/Mixtral-8x22B-Instruct-v0.1", allow_patterns=["params.json", "consolidated.safetensors", "tokenizer.model.v3"], local_dir=mistral_models_path)
+```
+
+### Chat
+
+After installing `mistral_inference`, a `mistral-chat` CLI command should be available in your environment. You can chat with the model using
+
+```
+mistral-chat $HOME/mistral_models/8x22B-Instruct-v0.1 --instruct --max_tokens 256
+```
+
+### Instruct following
 
-## Encode and Decode with `mistral_common`
-            
 ```py
+from mistral_inference.model import Transformer
+from mistral_inference.generate import generate
+
 from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
 from mistral_common.protocol.instruct.messages import UserMessage
 from mistral_common.protocol.instruct.request import ChatCompletionRequest
- 
-mistral_models_path = "MISTRAL_MODELS_PATH"
- 
-tokenizer = MistralTokenizer.v3()
- 
+
+
+tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
+# tokenizer = MistralTokenizer.v3()
+
+model = Transformer.from_folder(mistral_models_path)
+
 completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])
- 
+
 tokens = tokenizer.encode_chat_completion(completion_request).tokens
-```
- 
-## Inference with `mistral_inference`
- 
- ```py
-from mistral_inference.model import Transformer
-from mistral_inference.generate import generate
- 
-model = Transformer.from_folder(mistral_models_path)
-out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
 
-result = tokenizer.decode(out_tokens[0])
+out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
 
 print(result)
 ```
 
-## Inference with hugging face `transformers`
- 
-```py
-from transformers import AutoModelForCausalLM
- 
-model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
-model.to("cuda")
- 
-generated_ids = model.generate(tokens, max_new_tokens=1000, do_sample=True)
+### Function calling
 
-# decode with mistral tokenizer
-result = tokenizer.decode(generated_ids[0].tolist())
-print(result)
-```
+```py
+from mistral_common.protocol.instruct.tool_calls import Function, Tool
+from mistral_inference.model import Transformer
+from mistral_inference.generate import generate
 
-> [!TIP]
-> PRs to correct the `transformers` tokenizer so that it gives 1-to-1 the same results as the `mistral_common` reference implementation are very welcome!
-            
----
-The Mixtral-8x22B-Instruct-v0.1 Large Language Model (LLM) is an instruct fine-tuned version of the [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1).
-
-## Run the model 
-```python
-from transformers import AutoModelForCausalLM
-from mistral_common.protocol.instruct.messages import (
-    AssistantMessage,
-    UserMessage,
-)
-from mistral_common.protocol.instruct.tool_calls import (
-    Tool,
-    Function,
-)
 from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
-from mistral_common.tokens.instruct.normalize import ChatCompletionRequest
+from mistral_common.protocol.instruct.messages import UserMessage
+from mistral_common.protocol.instruct.request import ChatCompletionRequest
+
 
-device = "cuda" # the device to load the model onto
+tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
+# tokenizer = MistralTokenizer.v3()
 
-tokenizer_v3 = MistralTokenizer.v3()
+model = Transformer.from_folder(mistral_models_path)
 
-mistral_query = ChatCompletionRequest(
+completion_request = ChatCompletionRequest(
     tools=[
         Tool(
             function=Function(
@@ -105,126 +120,73 @@ mistral_query = ChatCompletionRequest(
         )
     ],
     messages=[
-        UserMessage(content="What's the weather like today in Paris"),
-    ],
-    model="test",
+        UserMessage(content="What's the weather like today in Paris?"),
+        ],
 )
 
-encodeds = tokenizer_v3.encode_chat_completion(mistral_query).tokens
-model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
-model_inputs = encodeds.to(device)
-model.to(device)
+tokens = tokenizer.encode_chat_completion(completion_request).tokens
+
+out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
 
-generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
-sp_tokenizer = tokenizer_v3.instruct_tokenizer.tokenizer
-decoded = sp_tokenizer.decode(generated_ids[0])
-print(decoded)
+print(result)
+```
+
+## Generate with `transformers`
+
+### Install dependencies
 ```
-Alternatively, you can run this example with the Hugging Face tokenizer.
-To use this example, you'll need transformers version 4.39.0 or higher.
-```console
-pip install transformers==4.39.0
+pip install transformers
 ```
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "mistralai/Mixtral-8x22B-Instruct-v0.1"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-conversation=[
-    {"role": "user", "content": "What's the weather like in Paris?"},
-    {
-        "role": "tool_calls",
-        "content": [
-            {
-                "name": "get_current_weather",
-                "arguments": {"location": "Paris, France", "format": "celsius"},
-                
-            }
-        ]
-    },
-    {
-        "role": "tool_results",
-        "content": {"content": 22}
-    },
-    {"role": "assistant", "content": "The current temperature in Paris, France is 22 degrees Celsius."},
-    {"role": "user", "content": "What about San Francisco?"}
-]
+### Instruct following
 
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+ 
+model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
+model.to("cuda")
+
+tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
 
-tools = [{"type": "function", "function": {"name":"get_current_weather", "description": "Get▁the▁current▁weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}},"required":["location","format"]}}}]
+messages = [
+    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
+    {"role": "user", "content": "Who are you?"},
+]
 
-# render the tool use prompt as a string:
-tool_use_prompt = tokenizer.apply_chat_template(
-            conversation,
-            chat_template="tool_use",
-            tools=tools,
+messages_prompt = tokenizer.apply_chat_template(
+            messages,
             tokenize=False,
             add_generation_prompt=True,
-
 )
-model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x22B-Instruct-v0.1")
 
 inputs = tokenizer(tool_use_prompt, return_tensors="pt")
 
-outputs = model.generate(**inputs, max_new_tokens=20)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
-
-# Instruct tokenizer
-The HuggingFace tokenizer included in this release should match our own. To compare: 
-`pip install mistral-common`
+outputs = model.generate(**inputs, max_new_tokens=1000)
+result = tokenizer.decode(outputs[0], skip_special_tokens=True)
 
+print(result)
+```
+Or:
 ```py
-from mistral_common.protocol.instruct.messages import (
-    AssistantMessage,
-    UserMessage,
-)
-from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
-from mistral_common.tokens.instruct.normalize import ChatCompletionRequest
+from transformers import pipeline
 
-from transformers import AutoTokenizer
-
-tokenizer_v3 = MistralTokenizer.v3()
+messages = [
+    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
+    {"role": "user", "content": "Who are you?"},
+]
+chatbot = pipeline("text-generation", model="mistralai/Mixtral-8x22B-Instruct-v0.1")
+result = chatbot(messages)
 
-mistral_query = ChatCompletionRequest(
-    messages=[
-        UserMessage(content="How many experts ?"),
-        AssistantMessage(content="8"),
-        UserMessage(content="How big ?"),
-        AssistantMessage(content="22B"),
-        UserMessage(content="Noice 🎉 !"),
-    ],
-    model="test",
-)
-hf_messages = mistral_query.model_dump()['messages']
+print(result)
+```
 
-tokenized_mistral = tokenizer_v3.encode_chat_completion(mistral_query).tokens
+## Limitations
 
-tokenizer_hf = AutoTokenizer.from_pretrained('mistralai/Mixtral-8x22B-Instruct-v0.1')
-tokenized_hf = tokenizer_hf.apply_chat_template(hf_messages, tokenize=True)
+The Mistral 8x22B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. 
+It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to
+make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.
 
-assert tokenized_hf == tokenized_mistral
-```
+## The Mistral AI Team
 
-# Function calling and special tokens
-This tokenizer includes more special tokens, related to function calling : 
-- [TOOL_CALLS]
-- [AVAILABLE_TOOLS]
-- [/AVAILABLE_TOOLS]
-- [TOOL_RESULTS]
-- [/TOOL_RESULTS]
-
-If you want to use this model with function calling, please be sure to apply it similarly to what is done in our [SentencePieceTokenizerV3](https://github.com/mistralai/mistral-common/blob/main/src/mistral_common/tokens/tokenizers/sentencepiece.py#L299).
-
-# The Mistral AI Team
-Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux,
-Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault,
-Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot,
-Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger,
-Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona,
-Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon,
-Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat,
-Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen,
-Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao,
-Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang,
-Valera Nemychnikova, William El Sayed, William Marshall
+Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang, Valera Nemychnikova, William El Sayed, William Marshall