  Most existing online knowledge distillation(OKD) techniques typically require
sophisticated modules to produce diverse knowledge for improving students'
generalization ability. In this paper, we strive to fully utilize multi-model
settings instead of well-designed modules to achieve a distillation effect with
excellent generalization performance. Generally, model generalization can be
reflected in the flatness of the loss landscape. Since averaging parameters of
multiple models can find flatter minima, we are inspired to extend the process
to the sampled convex combinations of multi-student models in OKD.
Specifically, by linearly weighting students' parameters in each training
batch, we construct a Hybrid-Weight Model(HWM) to represent the parameters
surrounding involved students. The supervision loss of HWM can estimate the
landscape's curvature of the whole region around students to measure the
generalization explicitly. Hence we integrate HWM's loss into students'
training and propose a novel OKD framework via parameter hybridization(OKDPH)
to promote flatter minima and obtain robust solutions. Considering the
redundancy of parameters could lead to the collapse of HWM, we further
introduce a fusion operation to keep the high similarity of students. Compared
to the state-of-the-art(SOTA) OKD methods and SOTA methods of seeking flat
minima, our OKDPH achieves higher performance with fewer parameters, benefiting
OKD with lightweight and robust characteristics. Our code is publicly available
at https://github.com/tianlizhang/OKDPH.
