  Despite the recent success of Graph Neural Networks, it remains challenging
to train a GNN on large graphs with millions of nodes and billions of edges,
which are prevalent in many graph-based applications. Traditional
sampling-based methods accelerate GNN training by dropping edges and nodes,
which impairs the graph integrity and model performance. Differently,
distributed GNN algorithms accelerate GNN training by utilizing multiple
computing devices and can be classified into two types: "partition-based"
methods enjoy low communication costs but suffer from information loss due to
dropped edges, while "propagation-based" methods avoid information loss but
suffer from prohibitive communication overhead caused by the neighbor
explosion. To jointly address these problems, this paper proposes DIGEST
(DIstributed Graph reprEsentation SynchronizaTion), a novel distributed GNN
training framework that synergizes the complementary strength of both
categories of existing methods. We propose to allow each device to utilize the
stale representations of its neighbors in other subgraphs during subgraph
parallel training. This way, our method preserves global graph information from
neighbors to avoid information loss and reduce communication costs. Our
convergence analysis demonstrates that DIGEST enjoys a state-of-the-art
convergence rate. Extensive experimental evaluation on large, real-world graph
datasets shows that DIGEST achieves up to 21.82 speedups without compromising
performance compared to state-of-the-art distributed GNN training frameworks.
