  Deep neural networks excel at learning the training data, but often provide
incorrect and confident predictions when evaluated on slightly different test
examples. This includes distribution shifts, outliers, and adversarial
examples. To address these issues, we propose Manifold Mixup, a simple
regularizer that encourages neural networks to predict less confidently on
interpolations of hidden representations. Manifold Mixup leverages semantic
interpolations as additional training signal, obtaining neural networks with
smoother decision boundaries at multiple levels of representation. As a result,
neural networks trained with Manifold Mixup learn class-representations with
fewer directions of variance. We prove theory on why this flattening happens
under ideal conditions, validate it on practical situations, and connect it to
previous works on information theory and generalization. In spite of incurring
no significant computation and being implemented in a few lines of code,
Manifold Mixup improves strong baselines in supervised learning, robustness to
single-step adversarial attacks, and test log-likelihood.
