  Graph Neural Networks (GNNs) offer a compact and computationally efficient
way to learn embeddings and classifications on graph data. GNN models are
frequently large, making distributed minibatch training necessary.
  The primary contribution of this paper is new methods for reducing
communication in the sampling step for distributed GNN training. Here, we
propose a matrix-based bulk sampling approach that expresses sampling as a
sparse matrix multiplication (SpGEMM) and samples multiple minibatches at once.
When the input graph topology does not fit on a single device, our method
distributes the graph and use communication-avoiding SpGEMM algorithms to scale
GNN minibatch sampling, enabling GNN training on much larger graphs than those
that can fit into a single device memory. When the input graph topology (but
not the embeddings) fits in the memory of one GPU, our approach (1) performs
sampling without communication, (2) amortizes the overheads of sampling a
minibatch, and (3) can represent multiple sampling algorithms by simply using
different matrix constructions. In addition to new methods for sampling, we
introduce a pipeline that uses our matrix-based bulk sampling approach to
provide end-to-end training results. We provide experimental results on the
largest Open Graph Benchmark (OGB) datasets on $128$ GPUs, and show that our
pipeline is $2.5\times$ faster than Quiver (a distributed extension to
PyTorch-Geometric) on a $3$-layer GraphSAGE network. On datasets outside of
OGB, we show a $8.46\times$ speedup on $128$ GPUs in per-epoch time. Finally,
we show scaling when the graph is distributed across GPUs and scaling for both
node-wise and layer-wise sampling algorithms.
