  Knowledge distillation which learns a lightweight student model by distilling
knowledge from a cumbersome teacher model is an attractive approach for
learning compact deep neural networks (DNNs). Recent works further improve
student network performance by leveraging multiple teacher networks. However,
most of the existing knowledge distillation-based multi-teacher methods use
separately pretrained teachers. This limits the collaborative learning between
teachers and the mutual learning between teachers and student. Network
quantization is another attractive approach for learning compact DNNs. However,
most existing network quantization methods are developed and evaluated without
considering multi-teacher support to enhance the performance of quantized
student model. In this paper, we propose a novel framework that leverages both
multi-teacher knowledge distillation and network quantization for learning low
bit-width DNNs. The proposed method encourages both collaborative learning
between quantized teachers and mutual learning between quantized teachers and
quantized student. During learning process, at corresponding layers, knowledge
from teachers will form an importance-aware shared knowledge which will be used
as input for teachers at subsequent layers and also be used to guide student.
Our experimental results on CIFAR100 and ImageNet datasets show that the
compact quantized student models trained with our method achieve competitive
results compared to other state-of-the-art methods, and in some cases, indeed
surpass the full precision models.
