  Graph Self-Supervised Learning (GSSL) provides a robust pathway for acquiring
embeddings without expert labelling, a capability that carries profound
implications for molecular graphs due to the staggering number of potential
molecules and the high cost of obtaining labels. However, GSSL methods are
designed not for optimisation within a specific domain but rather for
transferability across a variety of downstream tasks. This broad applicability
complicates their evaluation. Addressing this challenge, we present "Molecular
Graph Representation Evaluation" (MOLGRAPHEVAL), generating detailed profiles
of molecular graph embeddings with interpretable and diversified attributes.
MOLGRAPHEVAL offers a suite of probing tasks grouped into three categories: (i)
generic graph, (ii) molecular substructure, and (iii) embedding space
properties. By leveraging MOLGRAPHEVAL to benchmark existing GSSL methods
against both current downstream datasets and our suite of tasks, we uncover
significant inconsistencies between inferences drawn solely from existing
datasets and those derived from more nuanced probing. These findings suggest
that current evaluation methodologies fail to capture the entirety of the
landscape.
