  Large deep learning models have shown great potential for delivering
exceptional results in various applications. However, the training process can
be incredibly challenging due to the models' vast parameter sizes, often
consisting of hundreds of billions of parameters. Common distributed training
methods, such as data parallelism, tensor parallelism, and pipeline
parallelism, demand significant data communication throughout the process,
leading to prolonged wait times for some machines in physically distant
distributed systems. To address this issue, we propose a novel solution called
Hulk, which utilizes a modified graph neural network to optimize distributed
computing systems. Hulk not only optimizes data communication efficiency
between different countries or even different regions within the same city, but
also provides optimal distributed deployment of models in parallel. For
example, it can place certain layers on a machine in a specific region or pass
specific parameters of a model to a machine in a particular location. By using
Hulk in experiments, we were able to improve the time efficiency of training
large deep learning models on distributed systems by more than 20\%. Our open
source collection of unlabeled data:https://github.com/DLYuanGod/Hulk.
