  We introduce Adversarial Sparse Teacher (AST), a robust defense method
against distillation-based model stealing attacks. Our approach trains a
teacher model using adversarial examples to produce sparse logit responses and
increase the entropy of the output distribution. Typically, a model generates a
peak in its output corresponding to its prediction. By leveraging adversarial
examples, AST modifies the teacher model's original response, embedding a few
altered logits into the output while keeping the primary response slightly
higher. Concurrently, all remaining logits are elevated to further increase the
output distribution's entropy. All these complex manipulations are performed
using an optimization function with our proposed Exponential Predictive
Divergence (EPD) loss function. EPD allows us to maintain higher entropy levels
compared to traditional KL divergence, effectively confusing attackers.
Experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that AST outperforms
state-of-the-art methods, providing effective defense against model stealing
while preserving high accuracy. The source codes will be made publicly
available here soon.
