  While utilizing machine learning models, one of the most crucial aspects is
how bias and fairness affect model outcomes for diverse demographics. This
becomes especially relevant in the context of machine learning for medical
imaging applications as these models are increasingly being used for diagnosis
and treatment planning. In this paper, we study biases related to sex when
developing a machine learning model based on brain magnetic resonance images
(MRI). We investigate the effects of sex by performing brain age prediction
considering different experimental designs: model trained using only female
subjects, only male subjects and a balanced dataset. We also perform evaluation
on multiple MRI datasets (Calgary-Campinas(CC359) and CamCAN) to assess the
generalization capability of the proposed models. We found disparities in the
performance of brain age prediction models when trained on distinct sex
subgroups and datasets, in both final predictions and decision making (assessed
using interpretability models). Our results demonstrated variations in model
generalizability across sex-specific subgroups, suggesting potential biases in
models trained on unbalanced datasets. This underlines the critical role of
careful experimental design in generating fair and reliable outcomes.
