  Models with similar performances exhibit significant disagreement in the
predictions of individual samples, referred to as prediction churn. Our work
explores this phenomenon in graph neural networks by investigating differences
between models differing only in their initializations in their utilized
features for predictions. We propose a novel metric called Influence Difference
(ID) to quantify the variation in reasons used by nodes across models by
comparing their influence distribution. Additionally, we consider the
differences between nodes with a stable and an unstable prediction, positing
that both equally utilize different reasons and thus provide a meaningful
gradient signal to closely match two models even when the predictions for nodes
are similar. Based on our analysis, we propose to minimize this ID in Knowledge
Distillation, a domain where a new model should closely match an established
one. As an efficient approximation, we introduce DropDistillation (DD) that
matches the output for a graph perturbed by edge deletions. Our empirical
evaluation of six benchmark datasets for node classification validates the
differences in utilized features. DD outperforms previous methods regarding
prediction stability and overall performance in all considered Knowledge
Distillation experiments.
