  Deep neural networks bring in impressive accuracy in various applications,
but the success often relies on the heavy network architecture. Taking
well-trained heavy networks as teachers, classical teacher-student learning
paradigm aims to learn a student network that is lightweight yet accurate. In
this way, a portable student network with significantly fewer parameters can
achieve a considerable accuracy which is comparable to that of teacher network.
However, beyond accuracy, robustness of the learned student network against
perturbation is also essential for practical uses. Existing teacher-student
learning frameworks mainly focus on accuracy and compression ratios, but ignore
the robustness. In this paper, we make the student network produce more
confident predictions with the help of the teacher network, and analyze the
lower bound of the perturbation that will destroy the confidence of the student
network. Two important objectives regarding prediction scores and gradients of
examples are developed to maximize this lower bound, so as to enhance the
robustness of the student network without sacrificing the performance.
Experiments on benchmark datasets demonstrate the efficiency of the proposed
approach to learn robust student networks which have satisfying accuracy and
compact sizes.
