  Deep learning models tend to memorize training data, which hurts their
ability to generalize to under-represented classes. We empirically study a
convolutional neural network's internal representation of imbalanced image data
and measure the generalization gap between a model's feature embeddings in the
training and test sets, showing that the gap is wider for minority classes.
This insight enables us to design an efficient three-phase CNN training
framework for imbalanced data. The framework involves training the network
end-to-end on imbalanced data to learn accurate feature embeddings, performing
data augmentation in the learned embedded space to balance the train
distribution, and fine-tuning the classifier head on the embedded balanced
training data. We propose Expansive Over-Sampling (EOS) as a data augmentation
technique to utilize in the training framework. EOS forms synthetic training
instances as convex combinations between the minority class samples and their
nearest enemies in the embedded space to reduce the generalization gap. The
proposed framework improves the accuracy over leading cost-sensitive and
resampling methods commonly used in imbalanced learning. Moreover, it is more
computationally efficient than standard data pre-processing methods, such as
SMOTE and GAN-based oversampling, as it requires fewer parameters and less
training time.
