  Graph Neural Networks (GNNs) have shown success in many real-world
applications that involve graph-structured data. Most of the existing
single-node GNN training systems are capable of training medium-scale graphs
with tens of millions of edges; however, scaling them to large-scale graphs
with billions of edges remains challenging. In addition, it is challenging to
map GNN training algorithms onto a computation node as state-of-the-art
machines feature heterogeneous architecture consisting of multiple processors
and a variety of accelerators.
  We propose HyScale-GNN, a novel system to train GNN models on a single-node
heterogeneous architecture. HyScale- GNN performs hybrid training which
utilizes both the processors and the accelerators to train a model
collaboratively. Our system design overcomes the memory size limitation of
existing works and is optimized for training GNNs on large-scale graphs. We
propose a two-stage data pre-fetching scheme to reduce the communication
overhead during GNN training. To improve task mapping efficiency, we propose a
dynamic resource management mechanism, which adjusts the workload assignment
and resource allocation during runtime. We evaluate HyScale-GNN on a CPU-GPU
and a CPU-FPGA heterogeneous architecture. Using several large-scale datasets
and two widely-used GNN models, we compare the performance of our design with a
multi-GPU baseline implemented in PyTorch-Geometric. The CPU-GPU design and the
CPU-FPGA design achieve up to 2.08x speedup and 12.6x speedup, respectively.
Compared with the state-of-the-art large-scale multi-node GNN training systems
such as P3 and DistDGL, our CPU-FPGA design achieves up to 5.27x speedup using
a single node.
