  Mixed-precision quantization has been widely applied on deep neural networks
(DNNs) as it leads to significantly better efficiency-accuracy tradeoffs
compared to uniform quantization. Meanwhile, determining the exact precision of
each layer remains challenging. Previous attempts on bit-level regularization
and pruning-based dynamic precision adjustment during training suffer from
noisy gradients and unstable convergence. In this work, we propose Continuous
Sparsification Quantization (CSQ), a bit-level training method to search for
mixed-precision quantization schemes with improved stability. CSQ stabilizes
the bit-level mixed-precision training process with a bi-level gradual
continuous sparsification on both the bit values of the quantized weights and
the bit selection in determining the quantization precision of each layer. The
continuous sparsification scheme enables fully-differentiable training without
gradient approximation while achieving an exact quantized model in the end.A
budget-aware regularization of total model size enables the dynamic growth and
pruning of each layer's precision towards a mixed-precision quantization scheme
of the desired size. Extensive experiments show CSQ achieves better
efficiency-accuracy tradeoff than previous methods on multiple models and
datasets.
