  Knowledge distillation (KD) is an effective tool for compressing deep
classification models for edge devices. However, the performance of KD is
affected by the large capacity gap between the teacher and student networks.
Recent methods have resorted to a multiple teacher assistant (TA) setting for
KD, which sequentially decreases the size of the teacher model to relatively
bridge the size gap between these models. This paper proposes a new technique
called Curriculum Expert Selection for Knowledge Distillation (CES-KD) to
efficiently enhance the learning of a compact student under the capacity gap
problem. This technique is built upon the hypothesis that a student network
should be guided gradually using stratified teaching curriculum as it learns
easy (hard) data samples better and faster from a lower (higher) capacity
teacher network. Specifically, our method is a gradual TA-based KD technique
that selects a single teacher per input image based on a curriculum driven by
the difficulty in classifying the image. In this work, we empirically verify
our hypothesis and rigorously experiment with CIFAR-10, CIFAR-100, CINIC-10,
and ImageNet datasets and show improved accuracy on VGG-like models, ResNets,
and WideResNets architectures.
