  Despite recent advances in Graph Neural Networks (GNNs), their training
strategies remain largely under-explored. The conventional training strategy
learns over all nodes in the original graph(s) equally, which can be
sub-optimal as certain nodes are often more difficult to learn than others.
Here we present TuneUp, a simple curriculum-based training strategy for
improving the predictive performance of GNNs. TuneUp trains a GNN in two
stages. In the first stage, TuneUp applies conventional training to obtain a
strong base GNN. The base GNN tends to perform well on head nodes (nodes with
large degrees) but less so on tail nodes (nodes with small degrees). Therefore,
the second stage of TuneUp focuses on improving prediction on the difficult
tail nodes by further training the base GNN on synthetically generated tail
node data. We theoretically analyze TuneUp and show it provably improves
generalization performance on tail nodes. TuneUp is simple to implement and
applicable to a broad range of GNN architectures and prediction tasks.
Extensive evaluation of TuneUp on five diverse GNN architectures, three types
of prediction tasks, and both transductive and inductive settings shows that
TuneUp significantly improves the performance of the base GNN on tail nodes,
while often even improving the performance on head nodes. Altogether, TuneUp
produces up to 57.6% and 92.2% relative predictive performance improvement in
the transductive and the challenging inductive settings, respectively.
