  The error of an estimator can be decomposed into a (statistical) bias term, a
variance term, and an irreducible noise term. When we do bias analysis,
formally we are asking the question: "how good are the predictions?" The role
of bias in the error decomposition is clear: if we trust the labels/targets,
then we would want the estimator to have as low bias as possible, in order to
minimize error. Fair machine learning is concerned with the question: "Are the
predictions equally good for different demographic/social groups?" This has
naturally led to a variety of fairness metrics that compare some measure of
statistical bias on subsets corresponding to socially privileged and socially
disadvantaged groups. In this paper we propose a new family of performance
measures based on group-wise parity in variance. We demonstrate when group-wise
statistical bias analysis gives an incomplete picture, and what group-wise
variance analysis can tell us in settings that differ in the magnitude of
statistical bias. We develop and release an open-source library that reconciles
uncertainty quantification techniques with fairness analysis, and use it to
conduct an extensive empirical analysis of our variance-based fairness measures
on standard benchmarks.
