  Communication is a key bottleneck for distributed graph neural network (GNN)
training. This paper proposes GNNPipe, a new approach that scales the
distributed full-graph deep GNN training. Being the first to use layer-level
model parallelism for GNN training, GNNPipe partitions GNN layers among GPUs,
each device performs the computation for a disjoint subset of consecutive GNN
layers on the whole graph. Compared to graph parallelism with each GPU handling
a graph partition, GNNPipe reduces the communication volume by a factor of the
number of GNN layers. GNNPipe overcomes the unique challenges for pipelined
layer-level model parallelism on the whole graph by partitioning it into
dependent chunks, allowing the use of historical vertex embeddings, and
applying specific training techniques to ensure convergence. We also propose a
hybrid approach by combining GNNPipe with graph parallelism to handle large
graphs, achieve better computer resource utilization and ensure model
convergence. We build a general GNN training system supporting all three
parallelism setting. Extensive experiments show that our method reduces the
per-epoch training time by up to 2.45x (on average 1.58x) and reduces the
communication volume and overhead by up to 22.89x and 27.21x (on average 8.69x
and 11.60x), respectively, while achieving a comparable level of model accuracy
and convergence speed compared to graph parallelism.
