  Real data collected from different applications that have additional
topological structures and connection information are amenable to be
represented as a weighted graph. Considering the node labeling problem, Graph
Neural Networks (GNNs) is a powerful tool, which can mimic experts' decision on
node labeling. GNNs combine node features, connection patterns, and graph
structure by using a neural network to embed node information and pass it
through edges in the graph. We want to identify the patterns in the input data
used by the GNN model to make a decision and examine if the model works as we
desire. However, due to the complex data representation and non-linear
transformations, explaining decisions made by GNNs is challenging. In this
work, we propose new graph features' explanation methods to identify the
informative components and important node features. Besides, we propose a
pipeline to identify the key factors used for node classification. We use four
datasets (two synthetic and two real) to validate our methods. Our results
demonstrate that our explanation approach can mimic data patterns used for node
classification by human interpretation and disentangle different features in
the graphs. Furthermore, our explanation methods can be used for understanding
data, debugging GNN models, and examine model decisions.
