  Full batch training of Graph Convolutional Network (GCN) models is not
feasible on a single GPU for large graphs containing tens of millions of
vertices or more. Recent work has shown that, for the graphs used in the
machine learning community, communication becomes a bottleneck and scaling is
blocked outside of the single machine regime. Thus, we propose MG-GCN, a
multi-GPU GCN training framework taking advantage of the high-speed
communication links between the GPUs present in multi-GPU systems. MG-GCN
employs multiple High-Performance Computing optimizations, including efficient
re-use of memory buffers to reduce the memory footprint of training GNN models,
as well as communication and computation overlap. These optimizations enable
execution on larger datasets, that generally do not fit into memory of a single
GPU in state-of-the-art implementations. Furthermore, they contribute to
achieve superior speedup compared to the state-of-the-art. For example, MG-GCN
achieves super-linear speedup with respect to DGL, on the Reddit graph on both
DGX-1 (V100) and DGX-A100.
