  Data augmentation (DA) encodes invariance and provides implicit
regularization critical to a model's performance in image classification tasks.
However, while DA improves average accuracy, recent studies have shown that its
impact can be highly class dependent: achieving optimal average accuracy comes
at the cost of significantly hurting individual class accuracy by as much as
20% on ImageNet. There has been little progress in resolving class-level
accuracy drops due to a limited understanding of these effects. In this work,
we present a framework for understanding how DA interacts with class-level
learning dynamics. Using higher-quality multi-label annotations on ImageNet, we
systematically categorize the affected classes and find that the majority are
inherently ambiguous, co-occur, or involve fine-grained distinctions, while DA
controls the model's bias towards one of the closely related classes. While
many of the previously reported performance drops are explained by multi-label
annotations, our analysis of class confusions reveals other sources of accuracy
degradation. We show that simple class-conditional augmentation strategies
informed by our framework improve performance on the negatively affected
classes.
