  Ensemble knowledge distillation can extract knowledge from multiple teacher
models and encode it into a single student model. Many existing methods learn
and distill the student model on labeled data only. However, the teacher models
are usually learned on the same labeled data, and their predictions have high
correlations with groudtruth labels. Thus, they cannot provide sufficient
knowledge complementary to task labels for student teaching. Distilling on
unseen unlabeled data has the potential to enhance the knowledge transfer from
the teachers to the student. In this paper, we propose a unified and effective
ensemble knowledge distillation method that distills a single student model
from an ensemble of teacher models on both labeled and unlabeled data. Since
different teachers may have diverse prediction correctness on the same sample,
on labeled data we weight the predictions of different teachers according to
their correctness. In addition, we weight the distillation loss based on the
overall prediction correctness of the teacher ensemble to distill high-quality
knowledge. On unlabeled data, there is no groundtruth to evaluate prediction
correctness. Fortunately, the disagreement among teachers is an indication of
sample hardness, and thereby we weight the distillation loss based on teachers'
disagreement to emphasize knowledge distillation on important samples.
Extensive experiments on four datasets show the effectiveness of our proposed
ensemble distillation method.
