  Knowledge distillation (KD) is a successful approach for deep neural network
acceleration, with which a compact network (student) is trained by mimicking
the softmax output of a pre-trained high-capacity network (teacher). In
tradition, KD usually relies on access to the training samples and the
parameters of the white-box teacher to acquire the transferred knowledge.
However, these prerequisites are not always realistic due to storage costs or
privacy issues in real-world applications. Here we propose the concept of
decision-based black-box (DB3) knowledge distillation, with which the student
is trained by distilling the knowledge from a black-box teacher (parameters are
not accessible) that only returns classes rather than softmax outputs. We start
with the scenario when the training set is accessible. We represent a sample's
robustness against other classes by computing its distances to the teacher's
decision boundaries and use it to construct the soft label for each training
sample. After that, the student can be trained via standard KD. We then extend
this approach to a more challenging scenario in which even accessing the
training data is not feasible. We propose to generate pseudo samples
distinguished by the teacher's decision boundaries to the largest extent and
construct soft labels for them, which are used as the transfer set. We evaluate
our approaches on various benchmark networks and datasets and experiment
results demonstrate their effectiveness. Codes are available at:
https://github.com/zwang84/zsdb3kd.
