  In this paper, we propose a feature affinity (FA) assisted knowledge
distillation (KD) method to improve quantization-aware training of deep neural
networks (DNN). The FA loss on intermediate feature maps of DNNs plays the role
of teaching middle steps of a solution to a student instead of only giving
final answers in the conventional KD where the loss acts on the network logits
at the output level. Combining logit loss and FA loss, we found that the
quantized student network receives stronger supervision than from the labeled
ground-truth data. The resulting FAQD is capable of compressing model on
label-free data, which brings immediate practical benefits as pre-trained
teacher models are readily available and unlabeled data are abundant. In
contrast, data labeling is often laborious and expensive. Finally, we propose a
fast feature affinity (FFA) loss that accurately approximates FA loss with a
lower order of computational complexity, which helps speed up training for high
resolution image input.
