  Graph neural networks (GNNs) are highly effective on a variety of
graph-related tasks; however, they lack interpretability and transparency.
Current explainability approaches are typically local and treat GNNs as
black-boxes. They do not look inside the model, inhibiting human trust in the
model and explanations. Motivated by the ability of neurons to detect
high-level semantic concepts in vision models, we perform a novel analysis on
the behaviour of individual GNN neurons to answer questions about GNN
interpretability, and propose new metrics for evaluating the interpretability
of GNN neurons. We propose a novel approach for producing global explanations
for GNNs using neuron-level concepts to enable practitioners to have a
high-level view of the model. Specifically, (i) to the best of our knowledge,
this is the first work which shows that GNN neurons act as concept detectors
and have strong alignment with concepts formulated as logical compositions of
node degree and neighbourhood properties; (ii) we quantitatively assess the
importance of detected concepts, and identify a trade-off between training
duration and neuron-level interpretability; (iii) we demonstrate that our
global explainability approach has advantages over the current state-of-the-art
-- we can disentangle the explanation into individual interpretable concepts
backed by logical descriptions, which reduces potential for bias and improves
user-friendliness.
