  We propose the first unified theoretical analysis of mixed sample data
augmentation (MSDA), such as Mixup and CutMix. Our theoretical results show
that regardless of the choice of the mixing strategy, MSDA behaves as a
pixel-level regularization of the underlying training loss and a regularization
of the first layer parameters. Similarly, our theoretical results support that
the MSDA training strategy can improve adversarial robustness and
generalization compared to the vanilla training strategy. Using the theoretical
results, we provide a high-level understanding of how different design choices
of MSDA work differently. For example, we show that the most popular MSDA
methods, Mixup and CutMix, behave differently, e.g., CutMix regularizes the
input gradients by pixel distances, while Mixup regularizes the input gradients
regardless of pixel distances. Our theoretical results also show that the
optimal MSDA strategy depends on tasks, datasets, or model parameters. From
these observations, we propose generalized MSDAs, a Hybrid version of Mixup and
CutMix (HMix) and Gaussian Mixup (GMix), simple extensions of Mixup and CutMix.
Our implementation can leverage the advantages of Mixup and CutMix, while our
implementation is very efficient, and the computation cost is almost
neglectable as Mixup and CutMix. Our empirical study shows that our HMix and
GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and
ImageNet classification tasks. Source code is available at
https://github.com/naver-ai/hmix-gmix
