  The goal in label-imbalanced and group-sensitive classification is to
optimize relevant metrics such as balanced error and equal opportunity.
Classical methods, such as weighted cross-entropy, fail when training deep nets
to the terminal phase of training (TPT), that is training beyond zero training
error. This observation has motivated recent flurry of activity in developing
heuristic alternatives following the intuitive mechanism of promoting larger
margin for minorities. In contrast to previous heuristics, we follow a
principled analysis explaining how different loss adjustments affect margins.
First, we prove that for all linear classifiers trained in TPT, it is necessary
to introduce multiplicative, rather than additive, logit adjustments so that
the interclass margins change appropriately. To show this, we discover a
connection of the multiplicative CE modification to the cost-sensitive
support-vector machines. Perhaps counterintuitively, we also find that, at the
start of training, the same multiplicative weights can actually harm the
minority classes. Thus, while additive adjustments are ineffective in the TPT,
we show that they can speed up convergence by countering the initial negative
effect of the multiplicative weights. Motivated by these findings, we formulate
the vector-scaling (VS) loss, that captures existing techniques as special
cases. Moreover, we introduce a natural extension of the VS-loss to
group-sensitive classification, thus treating the two common types of
imbalances (label/group) in a unifying way. Importantly, our experiments on
state-of-the-art datasets are fully consistent with our theoretical insights
and confirm the superior performance of our algorithms. Finally, for imbalanced
Gaussian-mixtures data, we perform a generalization analysis, revealing
tradeoffs between balanced / standard error and equal opportunity.
