  We focus on the problem of training a deep neural network in generations. The
flowchart is that, in order to optimize the target network (student), another
network (teacher) with the same architecture is first trained, and used to
provide part of supervision signals in the next stage. While this strategy
leads to a higher accuracy, many aspects (e.g., why teacher-student
optimization helps) still need further explorations.
  This paper studies this problem from a perspective of controlling the
strictness in training the teacher network. Existing approaches mostly used a
hard distribution (e.g., one-hot vectors) in training, leading to a strict
teacher which itself has a high accuracy, but we argue that the teacher needs
to be more tolerant, although this often implies a lower accuracy. The
implementation is very easy, with merely an extra loss term added to the
teacher network, facilitating a few secondary classes to emerge and complement
to the primary class. Consequently, the teacher provides a milder supervision
signal (a less peaked distribution), and makes it possible for the student to
learn from inter-class similarity and potentially lower the risk of
over-fitting. Experiments are performed on standard image classification tasks
(CIFAR100 and ILSVRC2012). Although the teacher network behaves less powerful,
the students show a persistent ability growth and eventually achieve higher
classification accuracies than other competitors. Model ensemble and transfer
feature extraction also verify the effectiveness of our approach.
