  Convolutional neural networks for visual recognition require large amounts of
training samples and usually benefit from data augmentation. This paper
proposes PatchMix, a data augmentation method that creates new samples by
composing patches from pairs of images in a grid-like pattern. These new
samples are assigned label scores that are proportional to the number of
patches borrowed from each image. We then add a set of additional losses at the
patch-level to regularize and to encourage good representations at both the
patch and image levels. A ResNet-50 model trained on ImageNet using PatchMix
exhibits superior transfer learning capabilities across a wide array of
benchmarks. Although PatchMix can rely on random pairings and random grid-like
patterns for mixing, we explore evolutionary search as a guiding strategy to
jointly discover optimal grid-like patterns and image pairings. For this
purpose, we conceive a fitness function that bypasses the need to re-train a
model to evaluate each possible choice. In this way, PatchMix outperforms a
base model on CIFAR-10 (+1.91), CIFAR-100 (+5.31), Tiny Imagenet (+3.52), and
ImageNet (+1.16).
