  Despite advances in the field of Graph Neural Networks (GNNs), only a small
number (~5) of datasets are currently used to evaluate new models. This
continued reliance on a handful of datasets provides minimal insight into the
performance differences between models, and is especially challenging for
industrial practitioners who are likely to have datasets which look very
different from those used as academic benchmarks. In the course of our work on
GNN infrastructure and open-source software at Google, we have sought to
develop improved benchmarks that are robust, tunable, scalable,and
generalizable. In this work we introduce GraphWorld, a novel methodology and
system for benchmarking GNN models on an arbitrarily-large population of
synthetic graphs for any conceivable GNN task. GraphWorld allows a user to
efficiently generate a world with millions of statistically diverse datasets.
It is accessible, scalable, and easy to use. GraphWorld can be run on a single
machine without specialized hardware, or it can be easily scaled up to run on
arbitrary clusters or cloud frameworks. Using GraphWorld, a user has
fine-grained control over graph generator parameters, and can benchmark
arbitrary GNN models with built-in hyperparameter tuning. We present insights
from GraphWorld experiments regarding the performance characteristics of tens
of thousands of GNN models over millions of benchmark datasets. We further show
that GraphWorld efficiently explores regions of benchmark dataset space
uncovered by standard benchmarks, revealing comparisons between models that
have not been historically obtainable. Using GraphWorld, we also are able to
study in-detail the relationship between graph properties and task performance
metrics, which is nearly impossible with the classic collection of real-world
benchmarks.
