  Making fair decisions is crucial to ethically implementing machine learning
algorithms in social settings. In this work, we consider the celebrated
definition of counterfactual fairness [Kusner et al., NeurIPS, 2017]. We begin
by showing that an algorithm which satisfies counterfactual fairness also
satisfies demographic parity, a far simpler fairness constraint. Similarly, we
show that all algorithms satisfying demographic parity can be trivially
modified to satisfy counterfactual fairness. Together, our results indicate
that counterfactual fairness is basically equivalent to demographic parity,
which has important implications for the growing body of work on counterfactual
fairness. We then validate our theoretical findings empirically, analyzing
three existing algorithms for counterfactual fairness against three simple
benchmarks. We find that two simple benchmark algorithms outperform all three
existing algorithms -- in terms of fairness, accuracy, and efficiency -- on
several data sets. Our analysis leads us to formalize a concrete fairness goal:
to preserve the order of individuals within protected groups. We believe
transparency around the ordering of individuals within protected groups makes
fair algorithms more trustworthy. By design, the two simple benchmark
algorithms satisfy this goal while the existing algorithms for counterfactual
fairness do not.
