  Graph neural networks (GNN) have shown great success in learning from
graph-structured data. They are widely used in various applications, such as
recommendation, fraud detection, and search. In these domains, the graphs are
typically large and heterogeneous, containing many millions or billions of
vertices and edges of different types. To tackle this challenge, we develop
DistDGLv2, a system that extends DistDGL for training GNNs on massive
heterogeneous graphs in a mini-batch fashion, using distributed hybrid CPU/GPU
training. DistDGLv2 places graph data in distributed CPU memory and performs
mini-batch computation in GPUs. For ease of use, DistDGLv2 adopts API
compatible with Deep Graph Library (DGL)'s mini-batch training and
heterogeneous graph API, which enables distributed training with almost no code
modification. To ensure model accuracy, DistDGLv2 follows a synchronous
training approach and allows ego-networks forming mini-batches to include
non-local vertices. To ensure data locality and load balancing, DistDGLv2
partitions heterogeneous graphs by using a multi-level partitioning algorithm
with min-edge cut and multiple balancing constraints. DistDGLv2 deploys an
asynchronous mini-batch generation pipeline that makes computation and data
access asynchronous to fully utilize all hardware (CPU, GPU, network, PCIe).
The combination allows DistDGLv2 to train high-quality models while achieving
high parallel efficiency and memory scalability. We demonstrate DistDGLv2 on
various GNN workloads. Our results show that DistDGLv2 achieves 2-3X speedup
over DistDGL and 18X speedup over Euler. It takes only 5-10 seconds to complete
an epoch on graphs with hundreds of millions of vertices on a cluster with 64
GPUs.
