  Knowledge distillation (KD) is an efficient approach to transfer the
knowledge from a large "teacher" network to a smaller "student" network.
Traditional KD methods require lots of labeled training samples and a white-box
teacher (parameters are accessible) to train a good student. However, these
resources are not always available in real-world applications. The distillation
process often happens at an external party side where we do not have access to
much data, and the teacher does not disclose its parameters due to security and
privacy concerns. To overcome these challenges, we propose a black-box few-shot
KD method to train the student with few unlabeled training samples and a
black-box teacher. Our main idea is to expand the training set by generating a
diverse set of out-of-distribution synthetic images using MixUp and a
conditional variational auto-encoder. These synthetic images along with their
labels obtained from the teacher are used to train the student. We conduct
extensive experiments to show that our method significantly outperforms recent
SOTA few/zero-shot KD methods on image classification tasks. The code and
models are available at: https://github.com/nphdang/FS-BBT
