  Data practices shape research and practice on fairness in machine learning
(fair ML). Critical data studies offer important reflections and critiques for
the responsible advancement of the field by highlighting shortcomings and
proposing recommendations for improvement. In this work, we present a
comprehensive analysis of fair ML datasets, demonstrating how unreflective yet
common practices hinder the reach and reliability of algorithmic fairness
findings. We systematically study protected information encoded in tabular
datasets and their usage in 280 experiments across 142 publications.
  Our analyses identify three main areas of concern: (1) a \textbf{lack of
representation for certain protected attributes} in both data and evaluations;
(2) the widespread \textbf{exclusion of minorities} during data preprocessing;
and (3) \textbf{opaque data processing} threatening the generalization of
fairness research. By conducting exemplary analyses on the utilization of
prominent datasets, we demonstrate how unreflective data decisions
disproportionately affect minority groups, fairness metrics, and resultant
model comparisons. Additionally, we identify supplementary factors such as
limitations in publicly available data, privacy considerations, and a general
lack of awareness, which exacerbate these challenges. To address these issues,
we propose a set of recommendations for data usage in fairness research
centered on transparency and responsible inclusion. This study underscores the
need for a critical reevaluation of data practices in fair ML and offers
directions to improve both the sourcing and usage of datasets.
