  Pre-trained Transformer models such as BERT have shown great success in a
wide range of applications, but at the cost of substantial increases in model
complexity. Quantization-aware training (QAT) is a promising method to lower
the implementation cost and energy consumption. However, aggressive
quantization below 2-bit causes considerable accuracy degradation due to
unstable convergence, especially when the downstream dataset is not abundant.
This work proposes a proactive knowledge distillation method called Teacher
Intervention (TI) for fast converging QAT of ultra-low precision pre-trained
Transformers. TI intervenes layer-wise signal propagation with the intact
signal from the teacher to remove the interference of propagated quantization
errors, smoothing loss surface of QAT and expediting the convergence.
Furthermore, we propose a gradual intervention mechanism to stabilize the
recovery of subsections of Transformer layers from quantization. The proposed
schemes enable fast convergence of QAT and improve the model accuracy
regardless of the diverse characteristics of downstream fine-tuning tasks. We
demonstrate that TI consistently achieves superior accuracy with significantly
lower fine-tuning iterations on well-known Transformers of natural language
processing as well as computer vision compared to the state-of-the-art QAT
methods.
