  A multitude of work has shown that machine learning-based medical diagnosis
systems can be biased against certain subgroups of people. This has motivated a
growing number of bias mitigation algorithms that aim to address fairness
issues in machine learning. However, it is difficult to compare their
effectiveness in medical imaging for two reasons. First, there is little
consensus on the criteria to assess fairness. Second, existing bias mitigation
algorithms are developed under different settings, e.g., datasets, model
selection strategies, backbones, and fairness metrics, making a direct
comparison and evaluation based on existing results impossible. In this work,
we introduce MEDFAIR, a framework to benchmark the fairness of machine learning
models for medical imaging. MEDFAIR covers eleven algorithms from various
categories, nine datasets from different imaging modalities, and three model
selection criteria. Through extensive experiments, we find that the
under-studied issue of model selection criterion can have a significant impact
on fairness outcomes; while in contrast, state-of-the-art bias mitigation
algorithms do not significantly improve fairness outcomes over empirical risk
minimization (ERM) in both in-distribution and out-of-distribution settings. We
evaluate fairness from various perspectives and make recommendations for
different medical application scenarios that require different ethical
principles. Our framework provides a reproducible and easy-to-use entry point
for the development and evaluation of future bias mitigation algorithms in deep
learning. Code is available at https://github.com/ys-zong/MEDFAIR.
