  Knowledge distillation (KD) is an effective model compression technique where
a compact student network is taught to mimic the behavior of a complex and
highly trained teacher network. In contrast, Mutual Learning (ML) provides an
alternative strategy where multiple simple student networks benefit from
sharing knowledge, even in the absence of a powerful but static teacher
network. Motivated by these findings, we propose a single-teacher,
multi-student framework that leverages both KD and ML to achieve better
performance. Furthermore, an online distillation strategy is utilized to train
the teacher and students simultaneously. To evaluate the performance of the
proposed approach, extensive experiments were conducted using three different
versions of teacher-student networks on benchmark biomedical classification
(MSI vs. MSS) and object detection (Polyp Detection) tasks. Ensemble of student
networks trained in the proposed manner achieved better results than the
ensemble of students trained using KD or ML individually, establishing the
benefit of augmenting knowledge transfer from teacher to students with
peer-to-peer learning between students.
