  In knowledge distillation, previous feature distillation methods mainly focus
on the design of loss functions and the selection of the distilled layers,
while the effect of the feature projector between the student and the teacher
remains under-explored. In this paper, we first discuss a plausible mechanism
of the projector with empirical evidence and then propose a new feature
distillation method based on a projector ensemble for further performance
improvement. We observe that the student network benefits from a projector even
if the feature dimensions of the student and the teacher are the same. Training
a student backbone without a projector can be considered as a multi-task
learning process, namely achieving discriminative feature extraction for
classification and feature matching between the student and the teacher for
distillation at the same time. We hypothesize and empirically verify that
without a projector, the student network tends to overfit the teacher's feature
distributions despite having different architecture and weights initialization.
This leads to degradation on the quality of the student's deep features that
are eventually used in classification. Adding a projector, on the other hand,
disentangles the two learning tasks and helps the student network to focus
better on the main feature extraction task while still being able to utilize
teacher features as a guidance through the projector. Motivated by the positive
effect of the projector in feature distillation, we propose an ensemble of
projectors to further improve the quality of student features. Experimental
results on different datasets with a series of teacher-student pairs illustrate
the effectiveness of the proposed method.
