  Class-imbalance is one of the major challenges in real world datasets, where
a few classes (called majority classes) constitute much more data samples than
the rest (called minority classes). Learning deep neural networks using such
datasets leads to performances that are typically biased towards the majority
classes. Most of the prior works try to solve class-imbalance by assigning more
weights to the minority classes in various manners (e.g., data re-sampling,
cost-sensitive learning). However, we argue that the number of available
training data may not be always a good clue to determine the weighting strategy
because some of the minority classes might be sufficiently represented even by
a small number of training data. Overweighting samples of such classes can lead
to drop in the model's overall performance. We claim that the 'difficulty' of a
class as perceived by the model is more important to determine the weighting.
In this light, we propose a novel loss function named Class-wise
Difficulty-Balanced loss, or CDB loss, which dynamically distributes weights to
each sample according to the difficulty of the class that the sample belongs
to. Note that the assigned weights dynamically change as the 'difficulty' for
the model may change with the learning progress. Extensive experiments are
conducted on both image (artificially induced class-imbalanced MNIST,
long-tailed CIFAR and ImageNet-LT) and video (EGTEA) datasets. The results show
that CDB loss consistently outperforms the recently proposed loss functions on
class-imbalanced datasets irrespective of the data type (i.e., video or image).
