  Mixup, which creates synthetic training instances by linearly interpolating
random sample pairs, is a simple and yet effective regularization technique to
boost the performance of deep models trained with SGD. In this work, we report
a previously unobserved phenomenon in Mixup training: on a number of standard
datasets, the performance of Mixup-trained models starts to decay after
training for a large number of epochs, giving rise to a U-shaped generalization
curve. This behavior is further aggravated when the size of original dataset is
reduced. To help understand such a behavior of Mixup, we show theoretically
that Mixup training may introduce undesired data-dependent label noises to the
synthesized data. Via analyzing a least-square regression problem with a random
feature model, we explain why noisy labels may cause the U-shaped curve to
occur: Mixup improves generalization through fitting the clean patterns at the
early training stage, but as training progresses, Mixup becomes over-fitting to
the noise in the synthetic data. Extensive experiments are performed on a
variety of benchmark datasets, validating this explanation.
