  We study how to train a student deep neural network for visual recognition by
distilling knowledge from a blackbox teacher model in a data-efficient manner.
Progress on this problem can significantly reduce the dependence on large-scale
datasets for learning high-performing visual recognition models. There are two
major challenges. One is that the number of queries into the teacher model
should be minimized to save computational and/or financial costs. The other is
that the number of images used for the knowledge distillation should be small;
otherwise, it violates our expectation of reducing the dependence on
large-scale datasets. To tackle these challenges, we propose an approach that
blends mixup and active learning. The former effectively augments the few
unlabeled images by a big pool of synthetic images sampled from the convex hull
of the original images, and the latter actively chooses from the pool hard
examples for the student neural network and query their labels from the teacher
model. We validate our approach with extensive experiments.
