Hugging Face's logo

ArtusDev
/
mistralai_Magistral-Small-2509-EXL3

vllm
mistral-common
exl3
Model card Files
xet
Community
ArtusDev commited on
Commit
b461321
·
verified ·
1 Parent(s): bcac32f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +233 -30
README.md CHANGED
@@ -33,39 +33,242 @@ tags:
33
  - mistral-common
34
  - exl3
35
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
- ## EXL3 Quants of unsloth/Magistral-Small-2509
38
 
39
- EXL3 quants of [unsloth/Magistral-Small-2509](https://huggingface.co/mistralai/Magistral-Small-2509) using <a href="https://github.com/turboderp-org/exllamav3/">exllamav3</a> for quantization.
 
 
 
 
 
 
 
 
40
 
41
- Based on the HF conversion of the base Magistral model by [unsloth](https://huggingface.co/unsloth): [unsloth/Magistral-Small-2509](https://huggingface.co/unsloth/Magistral-Small-2509)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
- ### Quants
44
- | Quant(Revision) | Bits per Weight | Head Bits |
45
- | -------- | ---------- | --------- |
46
- | [2.5_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/2.5bpw_H6) | 2.5 | 6 |
47
- | [3.0_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/3.0bpw_H6) | 3.0 | 6 |
48
- | [3.5_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/3.5bpw_H6) | 3.5 | 6 |
49
- | [4.0_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/4.0bpw_H6) | 4.0 | 6 |
50
- | [4.5_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/4.5bpw_H6) | 4.5 | 6 |
51
- | [5.0_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/5.0bpw_H6) | 5.0 | 6 |
52
- | [6.0_H6](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/6.0bpw_H6) | 6.0 | 6 |
53
- | [8.0_H8](https://huggingface.co/ArtusDev/unsloth_Magistral-Small-2509-EXL3/tree/8.0bpw_H8) | 8.0 | 8 |
 
 
 
54
 
55
- ### Downloading quants with huggingface-cli
 
 
 
 
 
 
 
 
56
 
57
- <details>
58
- <summary>Click to view download instructions</summary>
59
-
60
- Install hugginface-cli:
61
-
62
- ```bash
63
- pip install -U "huggingface_hub[cli]"
64
- ```
65
-
66
- Download quant by targeting the specific quant revision (branch):
67
-
68
- ```
69
- huggingface-cli download ArtusDev/unsloth_Magistral-Small-2509-EXL3 --revision "5.0bpw_H6" --local-dir ./
70
- ```
71
- </details>
 
33
  - mistral-common
34
  - exl3
35
  ---
36
+ <style>
37
+ .container-dark {
38
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
39
+ line-height: 1.6;
40
+ color: #d4d4d4;
41
+ }
42
+ a {
43
+ color: #569cd6;
44
+ text-decoration: none;
45
+ font-weight: 600;
46
+ }
47
+ a:hover {
48
+ text-decoration: underline;
49
+ }
50
+ .card-dark {
51
+ background-color: #252526;
52
+ border-radius: 12px;
53
+ padding: 24px;
54
+ margin-bottom: 20px;
55
+ box-shadow: 0 4px 12px rgba(0,0,0,0.3);
56
+ border: 1px solid #3c3c3c;
57
+ }
58
+ .card-dark h1 {
59
+ font-size: 2.2em;
60
+ color: #ffffff;
61
+ text-align: center;
62
+ margin-bottom: 10px;
63
+ }
64
+ .card-dark.card-dark-title h1 {
65
+ font-size: 1.5em;
66
+ }
67
+ .card-dark .subtitle {
68
+ text-align: center;
69
+ font-size: 1.1em;
70
+ color: #a0a0a0;
71
+ }
72
+ .card-dark h2 {
73
+ font-size: 1.5em;
74
+ margin-top: 0;
75
+ padding-bottom: 10px;
76
+ border-bottom: 1px solid #3c3c3c;
77
+ color: #c586c0;
78
+ }
79
+ .card-dark h3 {
80
+ font-size: 1.2em;
81
+ color: #d4d4d4;
82
+ }
83
+ .styled-table {
84
+ display: table;
85
+ border: none;
86
+ width: 100%;
87
+ font-size: 0.95em;
88
+ margin-bottom: 0px;
89
+ }
90
+ .styled-table thead th {
91
+ background-color: #333333;
92
+ color: #c586c0;
93
+ text-align: left;
94
+ }
95
+ .styled-table th {
96
+ padding: 12px 15px;
97
+ }
98
+ .styled-table td {
99
+ padding: 0;
100
+ }
101
+ .styled-table table, .styled-table th, .styled-table td {
102
+ border-left: none;
103
+ border-right: none;
104
+ border-bottom: none;
105
+ }
106
+ .styled-table td {
107
+ border-bottom: 1px solid #3c3c3c;
108
+ }
109
+ .styled-table tbody tr {
110
+ transition: background-color 0.1s ease;
111
+ }
112
+ .styled-table tbody tr:hover {
113
+ background-color: #3a3a3a;
114
+ }
115
+ .styled-table tr:last-child td {
116
+ border-bottom: none;
117
+ }
118
+ .styled-table td a {
119
+ display: block;
120
+ padding: 12px 15px;
121
+ }
122
+ .styled-table td a.fake-link {
123
+ text-decoration:none;
124
+ color:inherit;
125
+ }
126
+ details {
127
+ margin-top: 20px;
128
+ border: 1px solid #3c3c3c;
129
+ border-radius: 8px;
130
+ overflow: hidden;
131
+ }
132
+ summary {
133
+ cursor: pointer;
134
+ padding: 12px 18px;
135
+ background-color: #6A5ACD;
136
+ font-weight: 600;
137
+ display: flex;
138
+ align-items: center;
139
+ gap: 10px;
140
+ justify-content: space-between;
141
+ list-style: none;
142
+ }
143
+ summary::-webkit-details-marker {
144
+ display: none;
145
+ }
146
+ summary:hover {
147
+ filter: brightness(1.1);
148
+ }
149
+ summary::after {
150
+ content: '';
151
+ display: inline-block;
152
+ width: 8px;
153
+ height: 8px;
154
+ border-bottom: 2px solid white;
155
+ border-right: 2px solid white;
156
+ transform: rotate(45deg);
157
+ transition: transform 0.3s ease;
158
+ }
159
+ details[open] > summary::after {
160
+ transform: rotate(225deg);
161
+ }
162
+ .details-content {
163
+ padding: 18px;
164
+ }
165
+ .btn-purple {
166
+ display: inline-block;
167
+ background-color: #6A5ACD;
168
+ color: white !important;
169
+ padding: 12px 24px;
170
+ border-radius: 8px;
171
+ text-decoration: none;
172
+ font-weight: 600;
173
+ transition: background-color 0.3s ease, transform 0.2s ease;
174
+ text-align: center;
175
+ }
176
+ .btn-purple:hover {
177
+ background-color: #7B68EE;
178
+ transform: translateY(-2px);
179
+ }
180
+ </style>
181
 
182
+ <div class="container-dark">
183
 
184
+ <div class="card-dark card-dark-title">
185
+ <h1>ArtusDev/mistralai_Magistral-Small-2509-EXL3</h1>
186
+ <p class="subtitle">
187
+ EXL3 quants of <a href="https://huggingface.co/mistralai/Magistral-Small-2509" target="_blank">mistralai/Magistral-Small-2509</a> using <a href="https://github.com/turboderp-org/exllamav3/" target="_blank">exllamav3</a> for quantization.
188
+ </p>
189
+ <p class="subtitle">
190
+ Based on the HF conversion of the base Magistral model by <a href="https://huggingface.co/unsloth" target="_blank">@unsloth</a>: <a href="https://huggingface.co/unsloth/Magistral-Small-2509" target="_blank">unsloth/Magistral-Small-2509</a>.
191
+ </p>
192
+ </div>
193
 
194
+ <div class="card-dark">
195
+ <h2>Quants</h2>
196
+ <table class="styled-table">
197
+ <thead>
198
+ <tr>
199
+ <th>Quant</th>
200
+ <th>BPW</th>
201
+ <th>Head Bits</th>
202
+ </tr>
203
+ </thead>
204
+ <tbody>
205
+ <tr>
206
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/2.5bpw_H6" target="_blank">2.5_H6</a></td>
207
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/2.5bpw_H6" target="_blank">2.5</a></td>
208
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/2.5bpw_H6" target="_blank">6</a></td>
209
+ </tr>
210
+ <tr>
211
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.0bpw_H6" target="_blank">3.0_H6</a></td>
212
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.0bpw_H6" target="_blank">3.0</a></td>
213
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.0bpw_H6" target="_blank">6</a></td>
214
+ </tr>
215
+ <tr>
216
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.5bpw_H6" target="_blank">3.5_H6</a></td>
217
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.5bpw_H6" target="_blank">3.5</a></td>
218
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/3.5bpw_H6" target="_blank">6</a></td>
219
+ </tr>
220
+ <tr>
221
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.0bpw_H6" target="_blank">4.0_H6</a></td>
222
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.0bpw_H6" target="_blank">4.0</a></td>
223
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.0bpw_H6" target="_blank">6</a></td>
224
+ </tr>
225
+ <tr>
226
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.5bpw_H6" target="_blank">4.5_H6</a></td>
227
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.5bpw_H6" target="_blank">4.5</a></td>
228
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/4.5bpw_H6" target="_blank">6</a></td>
229
+ </tr>
230
+ <tr>
231
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/5.0bpw_H6" target="_blank">5.0_H6</a></td>
232
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/5.0bpw_H6" target="_blank">5.0</a></td>
233
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/5.0bpw_H6" target="_blank">6</a></td>
234
+ </tr>
235
+ <tr>
236
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/6.0bpw_H6" target="_blank">6.0_H6</a></td>
237
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/6.0bpw_H6" target="_blank">6.0</a></td>
238
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/6.0bpw_H6" target="_blank">6</a></td>
239
+ </tr>
240
+ <tr>
241
+ <td><a href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/8.0bpw_H8" target="_blank">8.0_H8</a></td>
242
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/8.0bpw_H8" target="_blank">8.0</a></td>
243
+ <td><a class="fake-link" href="https://huggingface.co/ArtusDev/mistralai_Magistral-Small-2509-EXL3/tree/8.0bpw_H8" target="_blank">8</a></td>
244
+ </tr>
245
+ </tbody>
246
+ </table>
247
+ </div>
248
 
249
+ <div class="card-dark">
250
+ <h2>How to Download and Use Quants</h2>
251
+ <p>You can download quants by targeting specific size using the Hugging Face CLI.</p>
252
+ <details>
253
+ <summary>Click for download commands</summary>
254
+ <div class="details-content">
255
+ <b>1. Install huggingface-cli:</b>
256
+ <pre><code>pip install -U "huggingface_hub[cli]"</code></pre>
257
+ <b>2. Download a specific quant:</b>
258
+ <pre><code>huggingface-cli download ArtusDev/mistralai_Magistral-Small-2509-EXL3 --revision "5.0bpw_H6" --local-dir ./</code></pre>
259
+ </div>
260
+ </details>
261
+ <p>EXL3 quants can be run with any inference client that supports EXL3, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p>
262
+ </div>
263
 
264
+ <div class="card-dark">
265
+ <h2>Quant Requests</h2>
266
+ <div style="text-align: center; margin-top: 25px;">
267
+ <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quants</a>
268
+ </div>
269
+ <p class="subtitle">
270
+ See <a href="https://huggingface.co/ArtusDev/requests-exl" target="_blank">EXL community hub</a> for request guidelines.
271
+ </p>
272
+ </div>
273
 
274
+ </div>