  Significant computational resources are required to train Graph Neural
Networks (GNNs) at a large scale, and the process is highly data-intensive. One
of the most effective ways to reduce resource requirements is minibatch
training coupled with graph sampling. GNNs have the unique property that items
in a minibatch have overlapping data. However, the commonly implemented
Independent Minibatching approach assigns each Processing Element (PE) its own
minibatch to process, leading to duplicated computations and input data access
across PEs. This amplifies the Neighborhood Explosion Phenomenon (NEP), which
is the main bottleneck limiting scaling. To reduce the effects of NEP in the
multi-PE setting, we propose a new approach called Cooperative Minibatching.
Our approach capitalizes on the fact that the size of the sampled subgraph is a
concave function of the batch size, leading to significant reductions in the
amount of work per seed vertex as batch sizes increase. Hence, it is favorable
for processors equipped with a fast interconnect to work on a large minibatch
together as a single larger processor, instead of working on separate smaller
minibatches, even though global batch size is identical. We also show how to
take advantage of the same phenomenon in serial execution by generating
dependent consecutive minibatches. Our experimental evaluations show up to 4x
bandwidth savings for fetching vertex embeddings, by simply increasing this
dependency without harming model convergence. Combining our proposed
approaches, we achieve up to 64% speedup over Independent Minibatching on
single-node multi-GPU systems.
