  The graph convolutional network (GCN) is a go-to solution for machine
learning on graphs, but its training is notoriously difficult to scale both in
terms of graph size and the number of model parameters. Although some work has
explored training on large-scale graphs (e.g., GraphSAGE, ClusterGCN, etc.), we
pioneer efficient training of large-scale GCN models (i.e., ultra-wide,
overparameterized models) with the proposal of a novel, distributed training
framework. Our proposed training methodology, called GIST, disjointly
partitions the parameters of a GCN model into several, smaller sub-GCNs that
are trained independently and in parallel. In addition to being compatible with
all GCN architectures and existing sampling techniques for efficient GCN
training, GIST i) improves model performance, ii) scales to training on
arbitrarily large graphs, iii) decreases wall-clock training time, and iv)
enables the training of markedly overparameterized GCN models. Remarkably, with
GIST, we train an astonishgly-wide 32,768-dimensional GraphSAGE model, which
exceeds the capacity of a single GPU by a factor of 8x, to SOTA performance on
the Amazon2M dataset.
