  Mix-based augmentation has been proven fundamental to the generalization of
deep vision models. However, current augmentations only mix samples at the
current data batch during training, which ignores the possible knowledge
accumulated in the learning history. In this paper, we propose a recursive
mixed-sample learning paradigm, termed "RecursiveMix" (RM), by exploring a
novel training strategy that leverages the historical input-prediction-label
triplets. More specifically, we iteratively resize the input image batch from
the previous iteration and paste it into the current batch while their labels
are fused proportionally to the area of the operated patches. Further, a
consistency loss is introduced to align the identical image semantics across
the iterations, which helps the learning of scale-invariant feature
representations. Based on ResNet-50, RM largely improves classification
accuracy by $\sim$3.2\% on CIFAR100 and $\sim$2.8\% on ImageNet with negligible
extra computation/storage costs. In the downstream object detection task, the
RM pretrained model outperforms the baseline by 2.1 AP points and surpasses
CutMix by 1.4 AP points under the ATSS detector on COCO. In semantic
segmentation, RM also surpasses the baseline and CutMix by 1.9 and 1.1 mIoU
points under UperNet on ADE20K, respectively. Codes and pretrained models are
available at \url{https://github.com/megvii-research/RecursiveMix}.
