  Knowledge distillation has demonstrated encouraging performances in deep
model compression. Most existing approaches, however, require massive labeled
data to accomplish the knowledge transfer, making the model compression a
cumbersome and costly process. In this paper, we investigate the practical
few-shot knowledge distillation scenario, where we assume only a few samples
without human annotations are available for each category. To this end, we
introduce a principled dual-stage distillation scheme tailored for few-shot
data. In the first step, we graft the student blocks one by one onto the
teacher, and learn the parameters of the grafted block intertwined with those
of the other teacher blocks. In the second step, the trained student blocks are
progressively connected and then together grafted onto the teacher network,
allowing the learned student blocks to adapt themselves to each other and
eventually replace the teacher network. Experiments demonstrate that our
approach, with only a few unlabeled samples, achieves gratifying results on
CIFAR10, CIFAR100, and ILSVRC-2012. On CIFAR10 and CIFAR100, our performances
are even on par with those of knowledge distillation schemes that utilize the
full datasets. The source code is available at
https://github.com/zju-vipa/NetGraft.
