  Graph neural networks (GNNs) are quickly becoming the standard approach for
learning on graph structured data across several domains, but they lack
transparency in their decision-making. Several perturbation-based approaches
have been developed to provide insights into the decision making process of
GNNs. As this is an early research area, the methods and data used to evaluate
the generated explanations lack maturity. We explore these existing approaches
and identify common pitfalls in three main areas: (1) synthetic data generation
process, (2) evaluation metrics, and (3) the final presentation of the
explanation. For this purpose, we perform an empirical study to explore these
pitfalls along with their unintended consequences and propose remedies to
mitigate their effects.
