  The natural world often follows a long-tailed data distribution where only a
few classes account for most of the examples. This long-tail causes classifiers
to overfit to the majority class. To mitigate this, prior solutions commonly
adopt class rebalancing strategies such as data resampling and loss reshaping.
However, by treating each example within a class equally, these methods fail to
account for the important notion of example hardness, i.e., within each class
some examples are easier to classify than others. To incorporate this notion of
hardness into the learning process, we propose the EarLy-exiting
Framework(ELF). During training, ELF learns to early-exit easy examples through
auxiliary branches attached to a backbone network. This offers a dual
benefit-(1) the neural network increasingly focuses on hard examples, since
they contribute more to the overall network loss; and (2) it frees up
additional model capacity to distinguish difficult examples. Experimental
results on two large-scale datasets, ImageNet LT and iNaturalist'18,
demonstrate that ELF can improve state-of-the-art accuracy by more than 3
percent. This comes with the additional benefit of reducing up to 20 percent of
inference time FLOPS. ELF is complementary to prior work and can naturally
integrate with a variety of existing methods to tackle the challenge of
long-tailed distributions.
