  Graph Convolutional Networks (GCNs) is the state-of-the-art method for
learning graph-structured data, and training large-scale GCNs requires
distributed training across multiple accelerators such that each accelerator is
able to hold a partitioned subgraph. However, distributed GCN training incurs
prohibitive overhead of communicating node features and feature gradients among
partitions for every GCN layer during each training iteration, limiting the
achievable training efficiency and model scalability. To this end, we propose
PipeGCN, a simple yet effective scheme that hides the communication overhead by
pipelining inter-partition communication with intra-partition computation. It
is non-trivial to pipeline for efficient GCN training, as communicated node
features/gradients will become stale and thus can harm the convergence,
negating the pipeline benefit. Notably, little is known regarding the
convergence rate of GCN training with both stale features and stale feature
gradients. This work not only provides a theoretical convergence analysis but
also finds the convergence rate of PipeGCN to be close to that of the vanilla
distributed GCN training without any staleness. Furthermore, we develop a
smoothing method to further improve PipeGCN's convergence. Extensive
experiments show that PipeGCN can largely boost the training throughput
(1.7x~28.5x) while achieving the same accuracy as its vanilla counterpart and
existing full-graph training methods. The code is available at
https://github.com/RICE-EIC/PipeGCN.
