  Knowledge distillation as a broad class of methods has led to the development
of lightweight and memory efficient models, using a pre-trained model with a
large capacity (teacher network) to train a smaller model (student network).
Recently, additional variations for knowledge distillation, utilizing
activation maps of intermediate layers as the source of knowledge, have been
studied. Generally, in computer vision applications, it is seen that the
feature activation learned by a higher capacity model contains richer
knowledge, highlighting complete objects while focusing less on the background.
Based on this observation, we leverage the dual ability of the teacher to
accurately distinguish between positive (relevant to the target object) and
negative (irrelevant) areas. We propose a new loss function for distillation,
called angular margin-based distillation (AMD) loss. AMD loss uses the angular
distance between positive and negative features by projecting them onto a
hypersphere, motivated by the near angular distributions seen in many feature
extractors. Then, we create a more attentive feature that is angularly
distributed on the hypersphere by introducing an angular margin to the positive
feature. Transferring such knowledge from the teacher network enables the
student model to harness the higher discrimination of positive and negative
features for the teacher, thus distilling superior student models. The proposed
method is evaluated for various student-teacher network pairs on four public
datasets. Furthermore, we show that the proposed method has advantages in
compatibility with other learning techniques, such as using fine-grained
features, augmentation, and other distillation methods.
