  Exponential Moving Average (EMA) is a widely used weight averaging (WA)
regularization to learn flat optima for better generalizations without extra
cost in deep neural network (DNN) optimization. Despite achieving better
flatness, existing WA methods might fall into worse final performances or
require extra test-time computations. This work unveils the full potential of
EMA with a single line of modification, i.e., switching the EMA parameters to
the original model after each epoch, dubbed as Switch EMA (SEMA). From both
theoretical and empirical aspects, we demonstrate that SEMA can help DNNs to
reach generalization optima that better trade-off between flatness and
sharpness. To verify the effectiveness of SEMA, we conduct comparison
experiments with discriminative, generative, and regression tasks on vision and
language datasets, including image classification, self-supervised learning,
object detection and segmentation, image generation, video prediction,
attribute regression, and language modeling. Comprehensive results with popular
optimizers and networks show that SEMA is a free lunch for DNN training by
improving performances and boosting convergence speeds.
