  The expressive power of graph neural networks is usually measured by
comparing how many pairs of graphs or nodes an architecture can possibly
distinguish as non-isomorphic to those distinguishable by the $k$-dimensional
Weisfeiler-Leman ($k$-WL) test. In this paper, we uncover misalignments between
graph machine learning practitioners' conceptualizations of expressive power
and $k$-WL through a systematic analysis of the reliability and validity of
$k$-WL. We conduct a survey ($n = 18$) of practitioners to surface their
conceptualizations of expressive power and their assumptions about $k$-WL. In
contrast to practitioners' beliefs, our analysis (which draws from graph theory
and benchmark auditing) reveals that $k$-WL does not guarantee isometry, can be
irrelevant to real-world graph tasks, and may not promote generalization or
trustworthiness. We argue for extensional definitions and measurement of
expressive power based on benchmarks. We further contribute guiding questions
for constructing such benchmarks, which is critical for graph machine learning
practitioners to develop and transparently communicate our understandings of
expressive power.
