  Fair graph learning plays a pivotal role in numerous practical applications.
Recently, many fair graph learning methods have been proposed; however, their
evaluation often relies on poorly constructed semi-synthetic datasets or
substandard real-world datasets. In such cases, even a basic Multilayer
Perceptron (MLP) can outperform Graph Neural Networks (GNNs) in both utility
and fairness. In this work, we illustrate that many datasets fail to provide
meaningful information in the edges, which may challenge the necessity of using
graph structures in these problems. To address these issues, we develop and
introduce a collection of synthetic, semi-synthetic, and real-world datasets
that fulfill a broad spectrum of requirements. These datasets are thoughtfully
designed to include relevant graph structures and bias information crucial for
the fair evaluation of models. The proposed synthetic and semi-synthetic
datasets offer the flexibility to create data with controllable bias
parameters, thereby enabling the generation of desired datasets with
user-defined bias values with ease. Moreover, we conduct systematic evaluations
of these proposed datasets and establish a unified evaluation approach for fair
graph learning models. Our extensive experimental results with fair graph
learning methods across our datasets demonstrate their effectiveness in
benchmarking the performance of these methods. Our datasets and the code for
reproducing our experiments are available at
https://github.com/XweiQ/Benchmark-GraphFairness.
