  For on-device automatic speech recognition (ASR), quantization aware training
(QAT) is ubiquitous to achieve the trade-off between model predictive
performance and efficiency. Among existing QAT methods, one major drawback is
that the quantization centroids have to be predetermined and fixed. To overcome
this limitation, we introduce a regularization-free, "soft-to-hard" compression
mechanism with self-adjustable centroids in a mu-Law constrained space,
resulting in a simpler yet more versatile quantization scheme, called General
Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network
Transducer (RNN-T) and Conformer architectures on both LibriSpeech and
de-identified far-field datasets. Without accuracy degradation, GQ can compress
both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit
for fast and accurate inference. We observe a 30.73% memory footprint saving
and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical
device benchmarking.
