  Self-supervised learning (SSL) has become the de facto training paradigm of
large models, where pre-training is followed by supervised fine-tuning using
domain-specific data and labels. Despite demonstrating comparable performance
with supervised methods, comprehensive efforts to assess SSL's impact on
machine learning fairness (i.e., performing equally on different demographic
breakdowns) are lacking. Hypothesizing that SSL models would learn more
generic, hence less biased representations, this study explores the impact of
pre-training and fine-tuning strategies on fairness. We introduce a fairness
assessment framework for SSL, comprising five stages: defining dataset
requirements, pre-training, fine-tuning with gradual unfreezing, assessing
representation similarity conditioned on demographics, and establishing
domain-specific evaluation processes. We evaluate our method's generalizability
on three real-world human-centric datasets (i.e., MIMIC, MESA, and GLOBEM) by
systematically comparing hundreds of SSL and fine-tuned models on various
dimensions spanning from the intermediate representations to appropriate
evaluation metrics. Our findings demonstrate that SSL can significantly improve
model fairness, while maintaining performance on par with supervised
methods-exhibiting up to a 30% increase in fairness with minimal loss in
performance through self-supervision. We posit that such differences can be
attributed to representation dissimilarities found between the best- and the
worst-performing demographics across models-up to x13 greater for protected
attributes with larger performance discrepancies between segments.
