  Model smoothing is of central importance for obtaining a reliable teacher
model in the student-teacher framework, where the teacher generates surrogate
supervision signals to train the student. A popular model smoothing method is
the Temporal Moving Average (TMA), which continuously averages the teacher
parameters with the up-to-date student parameters. In this paper, we propose
"Spatial Ensemble", a novel model smoothing mechanism in parallel with TMA.
Spatial Ensemble randomly picks up a small fragment of the student model to
directly replace the corresponding fragment of the teacher model.
Consequentially, it stitches different fragments of historical student models
into a unity, yielding the "Spatial Ensemble" effect. Spatial Ensemble obtains
comparable student-teacher learning performance by itself and demonstrates
valuable complementarity with temporal moving average. Their integration, named
Spatial-Temporal Smoothing, brings general (sometimes significant) improvement
to the student-teacher learning framework on a variety of state-of-the-art
methods. For example, based on the self-supervised method BYOL, it yields +0.9%
top-1 accuracy improvement on ImageNet, while based on the semi-supervised
approach FixMatch, it increases the top-1 accuracy by around +6% on CIFAR-10
when only few training labels are available. Codes and models are available at:
https://github.com/tengteng95/Spatial_Ensemble.
