  Predictions over graphs play a crucial role in various domains, including
social networks and medicine. Graph Neural Networks (GNNs) have emerged as the
dominant approach for learning on graph data. Although a graph-structure is
provided as input to the GNN, in some cases the best solution can be obtained
by ignoring it. While GNNs have the ability to ignore the graph- structure in
such cases, it is not clear that they will. In this work, we show that GNNs
actually tend to overfit the given graph-structure. Namely, they use it even
when a better solution can be obtained by ignoring it. We analyze the implicit
bias of gradient-descent learning of GNNs and prove that when the ground truth
function does not use the graphs, GNNs are not guaranteed to learn a solution
that ignores the graph, even with infinite data. We examine this phenomenon
with respect to different graph distributions and find that regular graphs are
more robust to this over-fitting. We also prove that within the family of
regular graphs, GNNs are guaranteed to extrapolate when learning with gradient
descent. Finally, based on our empirical and theoretical findings, we
demonstrate on real-data how regular graphs can be leveraged to reduce graph
overfitting and enhance performance.
