  Knowledge distillation (KD) shows a bright promise as a powerful
regularization strategy to boost generalization ability by leveraging learned
sample-level soft targets. Yet, employing a complex pre-trained teacher network
or an ensemble of peer students in existing KD is both time-consuming and
computationally costly. Various self KD methods have been proposed to achieve
higher distillation efficiency. However, they either require extra network
architecture modification or are difficult to parallelize. To cope with these
challenges, we propose an efficient and reliable self-distillation framework,
named Self-Distillation from Last Mini-Batch (DLB). Specifically, we rearrange
the sequential sampling by constraining half of each mini-batch coinciding with
the previous iteration. Meanwhile, the rest half will coincide with the
upcoming iteration. Afterwards, the former half mini-batch distills on-the-fly
soft targets generated in the previous iteration. Our proposed mechanism guides
the training stability and consistency, resulting in robustness to label noise.
Moreover, our method is easy to implement, without taking up extra run-time
memory or requiring model structure modification. Experimental results on three
classification benchmarks illustrate that our approach can consistently
outperform state-of-the-art self-distillation approaches with different network
architectures. Additionally, our method shows strong compatibility with
augmentation strategies by gaining additional performance improvement. The code
is available at https://github.com/Meta-knowledge-Lab/DLB.
