  Graph neural network training is mainly categorized into mini-batch and
full-batch training methods. The mini-batch training method samples subgraphs
from the original graph in each iteration. This sampling operation introduces
extra computation overhead and reduces the training accuracy. Meanwhile, the
full-batch training method calculates the features and corresponding gradients
of all vertices in each iteration, and therefore has higher convergence
accuracy. However, in the distributed cluster, frequent remote accesses of
vertex features and gradients lead to huge communication overhead, thus
restricting the overall training efficiency.
  In this paper, we introduce the cached-based distributed full-batch graph
neural network training framework (CDFGNN). We propose the adaptive cache
mechanism to reduce the remote vertex access by caching the historical features
and gradients of neighbor vertices. Besides, we further optimize the
communication overhead by quantifying the messages and designing the graph
partition algorithm for the hierarchical communication architecture.
Experiments show that the adaptive cache mechanism reduces remote vertex
accesses by 63.14% on average. Combined with communication quantization and
hierarchical GP algorithm, CDFGNN outperforms the state-of-the-art distributed
full-batch training frameworks by 30.39% in our experiments. Our results
indicate that CDFGNN has great potential in accelerating distributed full-batch
GNN training tasks.
