nm-testing
/

TinyLlama-1.1B-compressed-tensors-kv-cache-scheme

Text Generation

text-generation-inference

compressed-tensors

Model card Files Files and versions

dtransposed commited on Jul 3, 2024

Commit

dfc9eed

·

verified ·

1 Parent(s): 941e856

Upload config.json with huggingface_hub

Files changed (1) hide show

config.json +2 -2

config.json CHANGED Viewed

@@ -16,7 +16,7 @@
           "num_bits": 8,
           "observer": "minmax",
           "observer_kwargs": {},
-          "strategy": "tensor",
           "symmetric": true,
           "type": "int"
         },
@@ -31,7 +31,7 @@
           "num_bits": 8,
           "observer": "minmax",
           "observer_kwargs": {},
-          "strategy": "tensor",
           "symmetric": true,
           "type": "int"
         }

           "num_bits": 8,
           "observer": "minmax",
           "observer_kwargs": {},
+          "strategy": "channel",
           "symmetric": true,
           "type": "int"
         },
           "num_bits": 8,
           "observer": "minmax",
           "observer_kwargs": {},
+          "strategy": "channel",
           "symmetric": true,
           "type": "int"
         }