  We present distributed algorithms for training dynamic Graph Neural Networks
(GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best
of our knowledge, this is the first scaling study on dynamic GNN. We devise
mechanisms for reducing the GPU memory usage and identify two execution time
bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting
properties of dynamic graphs, we design a graph difference-based strategy to
significantly reduce the transfer time. We develop a simple, but effective data
distribution technique under which the communication volume remains fixed and
linear in the input size, for any number of GPUs. Our experiments using
billion-size graphs on a system of 128 GPUs shows that: (i) the distribution
scheme achieves up to 30x speedup on 128 GPUs; (ii) the graph-difference
technique reduces the transfer time by a factor of up to 4.1x and the overall
execution time by up to 40%
