  Graph neural networks (GNNs), an emerging class of machine learning models
for graphs, have gained popularity for their superior performance in various
graph analytical tasks. Mini-batch training is commonly used to train GNNs on
large graphs, and data parallelism is the standard approach to scale mini-batch
training across multiple GPUs. One of the major performance costs in GNN
training is the loading of input features, which prevents GPUs from being fully
utilized. In this paper, we argue that this problem is exacerbated by
redundancies that are inherent to the data parallel approach. To address this
issue, we introduce a hybrid parallel mini-batch training paradigm called split
parallelism. Split parallelism avoids redundant data loads and splits the
sampling and training of each mini-batch across multiple GPUs online, at each
iteration, using a lightweight splitting algorithm. We implement split
parallelism in GSplit and show that it outperforms state-of-the-art mini-batch
training systems like DGL, Quiver, and $P^3$.
