  In this study, we systematically investigate the impact of class imbalance on
classification performance of convolutional neural networks (CNNs) and compare
frequently used methods to address the issue. Class imbalance is a common
problem that has been comprehensively studied in classical machine learning,
yet very limited systematic research is available in the context of deep
learning. In our study, we use three benchmark datasets of increasing
complexity, MNIST, CIFAR-10 and ImageNet, to investigate the effects of
imbalance on classification and perform an extensive comparison of several
methods to address the issue: oversampling, undersampling, two-phase training,
and thresholding that compensates for prior class probabilities. Our main
evaluation metric is area under the receiver operating characteristic curve
(ROC AUC) adjusted to multi-class tasks since overall accuracy metric is
associated with notable difficulties in the context of imbalanced data. Based
on results from our experiments we conclude that (i) the effect of class
imbalance on classification performance is detrimental; (ii) the method of
addressing class imbalance that emerged as dominant in almost all analyzed
scenarios was oversampling; (iii) oversampling should be applied to the level
that completely eliminates the imbalance, whereas the optimal undersampling
ratio depends on the extent of imbalance; (iv) as opposed to some classical
machine learning models, oversampling does not cause overfitting of CNNs; (v)
thresholding should be applied to compensate for prior class probabilities when
overall number of properly classified cases is of interest.
