  Variance in predictions across different trained models is a significant,
under-explored source of error in fair binary classification. In practice, the
variance on some data examples is so large that decisions can be effectively
arbitrary. To investigate this problem, we take an experimental approach and
make four overarching contributions: We: 1) Define a metric called
self-consistency, derived from variance, which we use as a proxy for measuring
and reducing arbitrariness; 2) Develop an ensembling algorithm that abstains
from classification when a prediction would be arbitrary; 3) Conduct the
largest to-date empirical study of the role of variance (vis-a-vis
self-consistency and arbitrariness) in fair binary classification; and, 4)
Release a toolkit that makes the US Home Mortgage Disclosure Act (HMDA)
datasets easily usable for future research. Altogether, our experiments reveal
shocking insights about the reliability of conclusions on benchmark datasets.
Most fair binary classification benchmarks are close-to-fair when taking into
account the amount of arbitrariness present in predictions -- before we even
try to apply any fairness interventions. This finding calls into question the
practical utility of common algorithmic fairness methods, and in turn suggests
that we should reconsider how we choose to measure fairness in binary
classification.
