  Online Knowledge Distillation (OKD) improves the involved models by
reciprocally exploiting the difference between teacher and student. Several
crucial bottlenecks over the gap between them -- e.g., Why and when does a
large gap harm the performance, especially for student? How to quantify the gap
between teacher and student? -- have received limited formal study. In this
paper, we propose Switchable Online Knowledge Distillation (SwitOKD), to answer
these questions. Instead of focusing on the accuracy gap at test phase by the
existing arts, the core idea of SwitOKD is to adaptively calibrate the gap at
training phase, namely distillation gap, via a switching strategy between two
modes -- expert mode (pause the teacher while keep the student learning) and
learning mode (restart the teacher). To possess an appropriate distillation
gap, we further devise an adaptive switching threshold, which provides a formal
criterion as to when to switch to learning mode or expert mode, and thus
improves the student's performance. Meanwhile, the teacher benefits from our
adaptive switching threshold and keeps basically on a par with other online
arts. We further extend SwitOKD to multiple networks with two basis topologies.
Finally, extensive experiments and analysis validate the merits of SwitOKD for
classification over the state-of-the-arts. Our code is available at
https://github.com/hfutqian/SwitOKD.
