  There is an increasing interest on accelerating neural networks for real-time
applications. We study the student-teacher strategy, in which a small and fast
student network is trained with the auxiliary information learned from a large
and accurate teacher network. We propose to use conditional adversarial
networks to learn the loss function to transfer knowledge from teacher to
student. The proposed method is particularly effective for relatively small
student networks. Moreover, experimental results show the effect of network
size when the modern networks are used as student. We empirically study the
trade-off between inference time and classification accuracy, and provide
suggestions on choosing a proper student network.
