  With a long history of traditional Graph Anomaly Detection (GAD) algorithms
and recently popular Graph Neural Networks (GNNs), it is still not clear (1)
how they perform under a standard comprehensive setting, (2) whether GNNs can
outperform traditional algorithms such as tree ensembles, and (3) how about
their efficiency on large-scale graphs. In response, we introduce GADBench -- a
benchmark tool dedicated to supervised anomalous node detection in static
graphs. GADBench facilitates a detailed comparison across 29 distinct models on
ten real-world GAD datasets, encompassing thousands to millions ($\sim$6M)
nodes. Our main finding is that tree ensembles with simple neighborhood
aggregation can outperform the latest GNNs tailored for the GAD task. We shed
light on the current progress of GAD, setting a robust groundwork for
subsequent investigations in this domain. GADBench is open-sourced at
https://github.com/squareRoot3/GADBench.
