  Addressing fairness in artificial intelligence (AI), particularly in medical
AI, is crucial for ensuring equitable healthcare outcomes. Recent efforts to
enhance fairness have introduced new methodologies and datasets in medical AI.
However, the fairness issue under the setting of domain transfer is almost
unexplored, while it is common that clinics rely on different imaging
technologies (e.g., different retinal imaging modalities) for patient
diagnosis. This paper presents FairDomain, a pioneering systemic study into
algorithmic fairness under domain shifts, employing state-of-the-art domain
adaptation (DA) and generalization (DG) algorithms for both medical
segmentation and classification tasks to understand how biases are transferred
between different domains. We also introduce a novel plug-and-play fair
identity attention (FIA) module that adapts to various DA and DG algorithms to
improve fairness by using self-attention to adjust feature importance based on
demographic attributes. Additionally, we curate the first fairness-focused
dataset with two paired imaging modalities for the same patient cohort on
medical segmentation and classification tasks, to rigorously assess fairness in
domain-shift scenarios. Excluding the confounding impact of demographic
distribution variation between source and target domains will allow clearer
quantification of the performance of domain transfer models. Our extensive
evaluations reveal that the proposed FIA significantly enhances both model
performance accounted for fairness across all domain shift settings (i.e., DA
and DG) with respect to different demographics, which outperforms existing
methods on both segmentation and classification. The code and data can be
accessed at https://ophai.hms.harvard.edu/datasets/harvard-fairdomain20k.
