  Data augmentation is widely applied and has shown its benefits in different
machine learning tasks. However, as recently observed in some downstream tasks,
data augmentation may introduce an unfair impact on classifications. While it
can improve the performance of some classes, it can actually be detrimental for
other classes, which can be problematic in some application domains. In this
paper, to counteract this phenomenon, we propose a FAir Classification approach
with a Two-player game (FACT). We first formulate the training of a classifier
with data augmentation as a fair optimization problem, which can be further
written as an adversarial two-player game. Following this formulation, we
propose a novel multiplicative weight optimization algorithm, for which we
theoretically prove that it can converge to a solution that is fair over
classes. Interestingly, our formulation also reveals that this fairness issue
over classes is not due to data augmentation only, but is in fact a general
phenomenon. Our empirical experiments demonstrate that the performance of our
learned classifiers is indeed more fairly distributed over classes in five
datasets, with only limited impact on the average accuracy.
