  As Graph Neural Networks (GNNs) become more pervasive, it becomes paramount
to build robust tools for computing explanations of their predictions. A key
desideratum is that these explanations are faithful, i.e., that they portray an
accurate picture of the GNN's reasoning process. A number of different
faithfulness metrics exist, begging the question of what faithfulness is
exactly, and what its properties are. We begin by showing that existing metrics
are not interchangeable -- i.e., explanations attaining high faithfulness
according to one metric may be unfaithful according to others -- and can be
systematically insensitive to important properties of the explanation, and
suggest how to address these issues. We proceed to show that, surprisingly,
optimizing for faithfulness is not always a sensible design goal. Specifically,
we show that for injective regular GNN architectures, perfectly faithful
explanations are completely uninformative. The situation is different for
modular GNNs, such as self-explainable and domain-invariant architectures,
where optimizing faithfulness does not compromise informativeness, and is also
unexpectedly tied to out-of-distribution generalization.
