  The choice of a graph learning (GL) model (i.e., a GL algorithm and its
hyperparameter settings) has a significant impact on the performance of
downstream tasks. However, selecting the right GL model becomes increasingly
difficult and time consuming as more and more GL models are developed.
Accordingly, it is of great significance and practical value to equip users of
GL with the ability to perform a near-instantaneous selection of an effective
GL model without manual intervention. Despite the recent attempts to tackle
this important problem, there has been no comprehensive benchmark environment
to evaluate the performance of GL model selection methods. To bridge this gap,
we present GLEMOS in this work, a comprehensive benchmark for instantaneous GL
model selection that makes the following contributions. (i) GLEMOS provides
extensive benchmark data for fundamental GL tasks, i.e., link prediction and
node classification, including the performances of 366 models on 457 graphs on
these tasks. (ii) GLEMOS designs multiple evaluation settings, and assesses how
effectively representative model selection techniques perform in these
different settings. (iii) GLEMOS is designed to be easily extended with new
models, new graphs, and new performance records. (iv) Based on the experimental
results, we discuss the limitations of existing approaches and highlight future
research directions. To promote research on this significant problem, we make
the benchmark data and code publicly available at
https://github.com/facebookresearch/glemos.
