  In this paper, we fully answer the above question through a key algebraic
condition on graph functions, called \textit{permutation compatibility}, that
relates permutations of weights and features of the graph to functional
constraints. We prove that: (i) a GNN, as a graph function, is necessarily
permutation compatible; (ii) conversely, any permutation compatible function,
when restricted on input graphs with distinct node features, can be generated
by a GNN; (iii) for arbitrary node features (not necessarily distinct), a
simple feature augmentation scheme suffices to generate a permutation
compatible function by a GNN; (iv) permutation compatibility can be verified by
checking only quadratically many functional constraints, rather than an
exhaustive search over all the permutations; (v) GNNs can generate \textit{any}
graph function once we augment the node features with node identities, thus
going beyond graph isomorphism and permutation compatibility. The above
characterizations pave the path to formally study the intricate connection
between GNNs and other algorithmic procedures on graphs. For instance, our
characterization implies that many natural graph problems, such as min-cut
value, max-flow value, max-clique size, and shortest path can be generated by a
GNN using a simple feature augmentation. In contrast, the celebrated
Weisfeiler-Lehman graph-isomorphism test fails whenever a permutation
compatible function with identical features cannot be generated by a GNN. At
the heart of our analysis lies a novel representation theorem that identifies
basis functions for GNNs. This enables us to translate the properties of the
target graph function into properties of the GNN's aggregation function.
