  We propose DiffQ a differentiable method for model compression for quantizing
model parameters without gradient approximations (e.g., Straight Through
Estimator). We suggest adding independent pseudo quantization noise to model
parameters during training to approximate the effect of a quantization
operator. DiffQ is differentiable both with respect to the unquantized weights
and the number of bits used. Given a single hyper-parameter balancing between
the quantized model size and accuracy, DiffQ optimizes the number of bits used
per individual weight or groups of weights, in end-to-end training. We
experimentally verify that our method is competitive with STE based
quantization techniques on several benchmarks and architectures for image
classification, language modeling, and audio source separation. For instance,
on the ImageNet dataset, DiffQ compresses a 12 layers transformer-based model
by more than a factor of 8, (lower than 4 bits precision per weight on
average), with a loss of 0.3% in model accuracy. Code is available at
github.com/facebookresearch/diffq.
