  The integration of machine learning models in various real-world applications
is becoming more prevalent to assist humans in their daily decision-making
tasks as a result of recent advancements in this field. However, it has been
discovered that there is a tradeoff between the accuracy and fairness of these
decision-making tasks. In some cases, these AI systems can be unfair by
exhibiting bias or discrimination against certain social groups, which can have
severe consequences in real life. Inspired by one of the most well-known human
learning skills called grouping, we address this issue by proposing a novel
machine learning framework where the ML model learns to group a diverse set of
problems into distinct subgroups to solve each subgroup using its specific
sub-model. Our proposed framework involves three stages of learning, which are
formulated as a three-level optimization problem: (i) learning to group
problems into different subgroups; (ii) learning group-specific sub-models for
problem-solving; and (iii) updating group assignments of training examples by
minimizing the validation loss. These three learning stages are performed
end-to-end in a joint manner using gradient descent. To improve fairness and
accuracy, we develop an efficient optimization algorithm to solve this
three-level optimization problem. To further reduce the risk of overfitting in
small datasets, we incorporate domain adaptation techniques in the second stage
of training. We further apply our method to neural architecture search.
Extensive experiments on various datasets demonstrate our method's
effectiveness and performance improvements in both fairness and accuracy. Our
proposed Learning by Grouping can reduce overfitting and achieve
state-of-the-art performances with fixed human-designed network architectures
and searchable network architectures on various datasets.
