  Class imbalance is a fundamental problem in computer vision applications such
as semantic segmentation. Specifically, uneven class distributions in a
training dataset often result in unsatisfactory performance on
under-represented classes. Many works have proposed to weight the standard
cross entropy loss function with pre-computed weights based on class
statistics, such as the number of samples and class margins. There are two
major drawbacks to these methods: 1) constantly up-weighting minority classes
can introduce excessive false positives in semantic segmentation; 2) a minority
class is not necessarily a hard class. The consequence is low precision due to
excessive false positives. In this regard, we propose a hard-class mining loss
by reshaping the vanilla cross entropy loss such that it weights the loss for
each class dynamically based on instantaneous recall performance. We show that
the novel recall loss changes gradually between the standard cross entropy loss
and the inverse frequency weighted loss. Recall loss also leads to improved
mean accuracy while offering competitive mean Intersection over Union (IoU)
performance. On Synthia dataset, recall loss achieves $9\%$ relative
improvement on mean accuracy with competitive mean IoU using DeepLab-ResNet18
compared to the cross entropy loss. Code available at
https://github.com/PotatoTian/recall-semseg.
