  Large scale graph optimization problems arise in many fields. This paper
presents an extensible, high performance framework (named OpenGraphGym-MG) that
uses deep reinforcement learning and graph embedding to solve large graph
optimization problems with multiple GPUs. The paper uses a common RL algorithm
(deep Q-learning) and a representative graph embedding (structure2vec) to
demonstrate the extensibility of the framework and, most importantly, to
illustrate the novel optimization techniques, such as spatial parallelism,
graph-level and node-level batched processing, distributed sparse graph
storage, efficient parallel RL training and inference algorithms, repeated
gradient descent iterations, and adaptive multiple-node selections. This study
performs a comprehensive performance analysis on parallel efficiency and memory
cost that proves the parallel RL training and inference algorithms are
efficient and highly scalable on a number of GPUs. This study also conducts a
range of large graph experiments, with both generated graphs (over 30 million
edges) and real-world graphs, using a single compute node (with six GPUs) of
the Summit supercomputer. Good scalability in both RL training and inference is
achieved: as the number of GPUs increases from one to six, the time of a single
step of RL training and a single step of RL inference on large graphs with more
than 30 million edges, is reduced from 316.4s to 54.5s, and 23.8s to 3.4s,
respectively. The research results on a single node lay out a solid foundation
for the future work to address graph optimization problems with a large number
of GPUs across multiple nodes in the Summit.
