  Artificial Intelligence (AI) finds widespread application across various
domains, but it sparks concerns about fairness in its deployment. The
prevailing discourse in classification often emphasizes outcome-based metrics
comparing sensitive subgroups without a nuanced consideration of the
differential impacts within subgroups. Bias mitigation techniques not only
affect the ranking of pairs of instances across sensitive groups, but often
also significantly affect the ranking of instances within these groups. Such
changes are hard to explain and raise concerns regarding the validity of the
intervention. Unfortunately, these effects remain under the radar in the
accuracy-fairness evaluation framework that is usually applied. Additionally,
we illustrate the effect of several popular bias mitigation methods, and how
their output often does not reflect real-world scenarios.
