  Graph classification benchmarks, vital for assessing and developing graph
neural networks (GNNs), have recently been scrutinized, as simple methods like
MLPs have demonstrated comparable performance. This leads to an important
question: Do these benchmarks effectively distinguish the advancements of GNNs
over other methodologies? If so, how do we quantitatively measure this
effectiveness? In response, we first propose an empirical protocol based on a
fair benchmarking framework to investigate the performance discrepancy between
simple methods and GNNs. We further propose a novel metric to quantify the
dataset effectiveness by considering both dataset complexity and model
performance. To the best of our knowledge, our work is the first to thoroughly
study and provide an explicit definition for dataset effectiveness in the graph
learning area. Through testing across 16 real-world datasets, we found our
metric to align with existing studies and intuitive assumptions. Finally, we
explore the causes behind the low effectiveness of certain datasets by
investigating the correlation between intrinsic graph properties and class
labels, and we developed a novel technique supporting the
correlation-controllable synthetic dataset generation. Our findings shed light
on the current understanding of benchmark datasets, and our new platform could
fuel the future evolution of graph classification benchmarks.
