  Data augmentation is a dominant method for reducing model overfitting and
improving generalization. Most existing data augmentation methods tend to find
a compromise in augmenting the data, \textit{i.e.}, increasing the amplitude of
augmentation carefully to avoid degrading some data too much and doing harm to
the model performance. We delve into the relationship between data augmentation
and model performance, revealing that the performance drop with heavy
augmentation comes from the presence of out-of-distribution (OOD) data.
Nonetheless, as the same data transformation has different effects for
different training samples, even for heavy augmentation, there remains part of
in-distribution data which is beneficial to model training. Based on the
observation, we propose a novel data augmentation method, named
\textbf{DualAug}, to keep the augmentation in distribution as much as possible
at a reasonable time and computational cost. We design a data mixing strategy
to fuse augmented data from both the basic- and the heavy-augmentation
branches. Extensive experiments on supervised image classification benchmarks
show that DualAug improve various automated data augmentation method. Moreover,
the experiments on semi-supervised learning and contrastive self-supervised
learning demonstrate that our DualAug can also improve related method. Code is
available at
\href{https://github.com/shuguang99/DualAug}{https://github.com/shuguang99/DualAug}.
