  Knowledge distillation (KD) is essentially a process of transferring a
teacher model's behavior, e.g., network response, to a student model. The
network response serves as additional supervision to formulate the machine
domain, which uses the data collected from the human domain as a transfer set.
Traditional KD methods hold an underlying assumption that the data collected in
both human domain and machine domain are both independent and identically
distributed (IID). We point out that this naive assumption is unrealistic and
there is indeed a transfer gap between the two domains. Although the gap offers
the student model external knowledge from the machine domain, the imbalanced
teacher knowledge would make us incorrectly estimate how much to transfer from
teacher to student per sample on the non-IID transfer set. To tackle this
challenge, we propose Inverse Probability Weighting Distillation (IPWD) that
estimates the propensity score of a training sample belonging to the machine
domain, and assigns its inverse amount to compensate for under-represented
samples. Experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of
IPWD for both two-stage distillation and one-stage self-distillation.
