  Knowledge distillation transfers knowledge from large models into small
models, and has recently made remarkable achievements. However, few studies has
investigated the mechanism of knowledge distillation against distribution
shift. Distribution shift refers to the data distribution drifts between
training and testing phases. In this paper, we reconsider the paradigm of
knowledge distillation by reformulating the objective function in shift
situations. Under the real scenarios, we propose a unified and systematic
framework to benchmark knowledge distillation against two general
distributional shifts including diversity and correlation shift. The evaluation
benchmark covers more than 30 methods from algorithmic, data-driven, and
optimization perspectives for five benchmark datasets. Overall, we conduct
extensive experiments on the student model. We reveal intriguing observations
of poor teaching performance under distribution shifts; in particular, complex
algorithms and data augmentation offer limited gains in many cases.
