  The current evaluation protocol of long-tailed visual recognition trains the
classification model on the long-tailed source label distribution and evaluates
its performance on the uniform target label distribution. Such protocol has
questionable practicality since the target may also be long-tailed. Therefore,
we formulate long-tailed visual recognition as a label shift problem where the
target and source label distributions are different. One of the significant
hurdles in dealing with the label shift problem is the entanglement between the
source label distribution and the model prediction. In this paper, we focus on
disentangling the source label distribution from the model prediction. We first
introduce a simple but overlooked baseline method that matches the target label
distribution by post-processing the model prediction trained by the
cross-entropy loss and the Softmax function. Although this method surpasses
state-of-the-art methods on benchmark datasets, it can be further improved by
directly disentangling the source label distribution from the model prediction
in the training phase. Thus, we propose a novel method, LAbel distribution
DisEntangling (LADE) loss based on the optimal bound of Donsker-Varadhan
representation. LADE achieves state-of-the-art performance on benchmark
datasets such as CIFAR-100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018.
Moreover, LADE outperforms existing methods on various shifted target label
distributions, showing the general adaptability of our proposed method.
