  Machine learning (ML) holds great promise for improving healthcare, but it is
critical to ensure that its use will not propagate or amplify health
disparities. An important step is to characterize the (un)fairness of ML models
- their tendency to perform differently across subgroups of the population -
and to understand its underlying mechanisms. One potential driver of
algorithmic unfairness, shortcut learning, arises when ML models base
predictions on improper correlations in the training data. However, diagnosing
this phenomenon is difficult, especially when sensitive attributes are causally
linked with disease. Using multi-task learning, we propose the first method to
assess and mitigate shortcut learning as a part of the fairness assessment of
clinical ML systems, and demonstrate its application to clinical tasks in
radiology and dermatology. Finally, our approach reveals instances when
shortcutting is not responsible for unfairness, highlighting the need for a
holistic approach to fairness mitigation in medical AI.
