  Fairness is a critical concern in deep learning, especially in healthcare,
where these models influence diagnoses and treatment decisions. Although
fairness has been investigated in the vision-only domain, the fairness of
medical vision-language (VL) models remains unexplored due to the scarcity of
medical VL datasets for studying fairness. To bridge this research gap, we
introduce the first fair vision-language medical dataset Harvard-FairVLMed that
provides detailed demographic attributes, ground-truth labels, and clinical
notes to facilitate an in-depth examination of fairness within VL foundation
models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis
of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and
medical domains, across four different protected attributes. Our results
highlight significant biases in all VL models, with Asian, Male, Non-Hispanic,
and Spanish being the preferred subgroups across the protected attributes of
race, gender, ethnicity, and language, respectively. In order to alleviate
these biases, we propose FairCLIP, an optimal-transport-based approach that
achieves a favorable trade-off between performance and fairness by reducing the
Sinkhorn distance between the overall sample distribution and the distributions
corresponding to each demographic group. As the first VL dataset of its kind,
Harvard-FairVLMed holds the potential to catalyze advancements in the
development of machine learning models that are both ethically aware and
clinically effective. Our dataset and code are available at
https://ophai.hms.harvard.edu/datasets/harvard-fairvlmed10k.
