  Structural concept complexity, class overlap, and data scarcity are some of
the most important factors influencing the performance of classifiers under
class imbalance conditions. When these effects were uncovered in the early
2000s, understandably, the classifiers on which they were demonstrated belonged
to the classical rather than Deep Learning categories of approaches. As Deep
Learning is gaining ground over classical machine learning and is beginning to
be used in critical applied settings, it is important to assess systematically
how well they respond to the kind of challenges their classical counterparts
have struggled with in the past two decades. The purpose of this paper is to
study the behavior of deep learning systems in settings that have previously
been deemed challenging to classical machine learning systems to find out
whether the depth of the systems is an asset in such settings. The results in
both artificial and real-world image datasets (MNIST Fashion, CIFAR-10) show
that these settings remain mostly challenging for Deep Learning systems and
that deeper architectures seem to help with structural concept complexity but
not with overlap challenges in simple artificial domains. Data scarcity is not
overcome by deeper layers, either. In the real-world image domains, where
overfitting is a greater concern than in the artificial domains, the advantage
of deeper architectures is less obvious: while it is observed in certain cases,
it is quickly cancelled as models get deeper and perform worse than their
shallower counterparts.
