  Imaging exams, such as chest radiography, will yield a small set of common
findings and a much larger set of uncommon findings. While a trained
radiologist can learn the visual presentation of rare conditions by studying a
few representative examples, teaching a machine to learn from such a
"long-tailed" distribution is much more difficult, as standard methods would be
easily biased toward the most frequent classes. In this paper, we present a
comprehensive benchmark study of the long-tailed learning problem in the
specific domain of thorax diseases on chest X-rays. We focus on learning from
naturally distributed chest X-ray data, optimizing classification accuracy over
not only the common "head" classes, but also the rare yet critical "tail"
classes. To accomplish this, we introduce a challenging new long-tailed chest
X-ray benchmark to facilitate research on developing long-tailed learning
methods for medical image classification. The benchmark consists of two chest
X-ray datasets for 19- and 20-way thorax disease classification, containing
classes with as many as 53,000 and as few as 7 labeled training images. We
evaluate both standard and state-of-the-art long-tailed learning methods on
this new benchmark, analyzing which aspects of these methods are most
beneficial for long-tailed medical image classification and summarizing
insights for future algorithm design. The datasets, trained models, and code
are available at https://github.com/VITA-Group/LongTailCXR.
