QwQ-32B-Preview-bnb-4bit

Introduction

QwQ-32B-Preview-bnb-4bit is a 4-bit quantized version of the QwQ-32B-Preview model, utilizing the Bits and Bytes (bnb) quantization technique. This quantization significantly reduces the model's size and inference latency, making it more accessible for deployment on resource-constrained hardware.

Model Details

• Quantization: 4-bit using Bits and Bytes (bnb)
• Base Model: Qwen/QwQ-32B-Preview
• Parameters: 32.5 billion
• Context Length: Up to 32,768 tokens