  Long-tailed instance segmentation is a challenging task due to the extreme
imbalance of training samples among classes. It causes severe biases of the
head classes (with majority samples) against the tailed ones. This renders "how
to appropriately define and alleviate the bias" one of the most important
issues. Prior works mainly use label distribution or mean score information to
indicate a coarse-grained bias. In this paper, we explore to excavate the
confusion matrix, which carries the fine-grained misclassification details, to
relieve the pairwise biases, generalizing the coarse one. To this end, we
propose a novel Pairwise Class Balance (PCB) method, built upon a confusion
matrix which is updated during training to accumulate the ongoing prediction
preferences. PCB generates fightback soft labels for regularization during
training. Besides, an iterative learning paradigm is developed to support a
progressive and smooth regularization in such debiasing. PCB can be plugged and
played to any existing method as a complement. Experimental results on LVIS
demonstrate that our method achieves state-of-the-art performance without bells
and whistles. Superior results across various architectures show the
generalization ability. The code and trained models are available at
https://github.com/megvii-research/PCB.
