  We propose an analysis in fair learning that preserves the utility of the
data while reducing prediction disparities under the criteria of group
sufficiency. We focus on the scenario where the data contains multiple or even
many subgroups, each with limited number of samples. As a result, we present a
principled method for learning a fair predictor for all subgroups via
formulating it as a bilevel objective. Specifically, the subgroup specific
predictors are learned in the lower-level through a small amount of data and
the fair predictor. In the upper-level, the fair predictor is updated to be
close to all subgroup specific predictors. We further prove that such a bilevel
objective can effectively control the group sufficiency and generalization
error. We evaluate the proposed framework on real-world datasets. Empirical
evidence suggests the consistently improved fair predictions, as well as the
comparable accuracy to the baselines.
