  Though machine learning algorithms excel at minimizing the average loss over
a population, this might lead to large discrepancies between the losses across
groups within the population. To capture this inequality, we introduce and
study a notion we call maximum weighted loss discrepancy (MWLD), the maximum
(weighted) difference between the loss of a group and the loss of the
population. We relate MWLD to group fairness notions and robustness to
demographic shifts. We then show MWLD satisfies the following three properties:
1) It is statistically impossible to estimate MWLD when all groups have equal
weights. 2) For a particular family of weighting functions, we can estimate
MWLD efficiently. 3) MWLD is related to loss variance, a quantity that arises
in generalization bounds. We estimate MWLD with different weighting functions
on four common datasets from the fairness literature. We finally show that loss
variance regularization can halve the loss variance of a classifier and hence
reduce MWLD without suffering a significant drop in accuracy.
