  Data augmentation is a widely used training trick in deep learning to improve
the network generalization ability. Despite many encouraging results, several
recent studies did point out limitations of the conventional data augmentation
scheme in certain scenarios, calling for a better theoretical understanding of
data augmentation. In this work, we develop a comprehensive analysis that
reveals pros and cons of data augmentation. The main limitation of data
augmentation arises from the data bias, i.e. the augmented data distribution
can be quite different from the original one. This data bias leads to a
suboptimal performance of existing data augmentation methods. To this end, we
develop two novel algorithms, termed "AugDrop" and "MixLoss", to correct the
data bias in the data augmentation. Our theoretical analysis shows that both
algorithms are guaranteed to improve the effect of data augmentation through
the bias correction, which is further validated by our empirical studies.
Finally, we propose a generic algorithm "WeMix" by combining AugDrop and
MixLoss, whose effectiveness is observed from extensive empirical evaluations.
