  Recently, researchers have shown an increased interest in the online
knowledge distillation. Adopting an one-stage and end-to-end training fashion,
online knowledge distillation uses aggregated intermediated predictions of
multiple peer models for training. However, the absence of a powerful teacher
model may result in the homogeneity problem between group peers, affecting the
effectiveness of group distillation adversely. In this paper, we propose a
novel online knowledge distillation method, \textbf{C}hannel
\textbf{S}elf-\textbf{S}upervision for Online Knowledge Distillation (CSS),
which structures diversity in terms of input, target, and network to alleviate
the homogenization problem. Specifically, we construct a dual-network
multi-branch structure and enhance inter-branch diversity through
self-supervised learning, adopting the feature-level transformation and
augmenting the corresponding labels. Meanwhile, the dual network structure has
a larger space of independent parameters to resist the homogenization problem
during distillation. Extensive quantitative experiments on CIFAR-100 illustrate
that our method provides greater diversity than OKDDip and we also give pretty
performance improvement, even over the state-of-the-art such as PCL. The
results on three fine-grained datasets (StanfordDogs, StanfordCars,
CUB-200-211) also show the significant generalization capability of our
approach.
