  Knowledge distillation is an effective paradigm for boosting the performance
of pocket-size model, especially when multiple teacher models are available,
the student would break the upper limit again. However, it is not economical to
train diverse teacher models for the disposable distillation. In this paper, we
introduce a new concept dubbed Avatars for distillation, which are the
inference ensemble models derived from the teacher. Concretely, (1) For each
iteration of distillation training, various Avatars are generated by a
perturbation transformation. We validate that Avatars own higher upper limit of
working capacity and teaching ability, aiding the student model in learning
diverse and receptive knowledge perspectives from the teacher model. (2) During
the distillation, we propose an uncertainty-aware factor from the variance of
statistical differences between the vanilla teacher and Avatars, to adjust
Avatars' contribution on knowledge transfer adaptively. Avatar Knowledge
Distillation AKD is fundamentally different from existing methods and refines
with the innovative view of unequal training. Comprehensive experiments
demonstrate the effectiveness of our Avatars mechanism, which polishes up the
state-of-the-art distillation methods for dense prediction without more extra
computational cost. The AKD brings at most 0.7 AP gains on COCO 2017 for Object
Detection and 1.83 mIoU gains on Cityscapes for Semantic Segmentation,
respectively. Code is available at https://github.com/Gumpest/AvatarKD.
