  Neural Networks can perform poorly when the training label distribution is
heavily imbalanced, as well as when the testing data differs from the training
distribution. In order to deal with shift in the testing label distribution,
which imbalance causes, we motivate the problem from the perspective of an
optimal Bayes classifier and derive a post-training prior rebalancing technique
that can be solved through a KL-divergence based optimization. This method
allows a flexible post-training hyper-parameter to be efficiently tuned on a
validation set and effectively modify the classifier margin to deal with this
imbalance. We further combine this method with existing likelihood shift
methods, re-interpreting them from the same Bayesian perspective, and
demonstrating that our method can deal with both problems in a unified way. The
resulting algorithm can be conveniently used on probabilistic classification
problems agnostic to underlying architectures. Our results on six different
datasets and five different architectures show state of art accuracy, including
on large-scale imbalanced datasets such as iNaturalist for classification and
Synthia for semantic segmentation. Please see
https://github.com/GT-RIPL/UNO-IC.git for implementation.
