  The development of large-scale models relies on a vast number of computing
resources. For example, the GPT-4 model (1.8 trillion parameters) requires
25000 A100 GPUs for its training. It is a challenge to build a large-scale
cluster with a type of GPU-accelerator. Using multiple types of
GPU-accelerators to construct a cluster is an effective way to solve the
problem of insufficient homogeneous GPU-accelerators. However, the existing
distributed training systems for large-scale models only support homogeneous
GPU-accelerators, not heterogeneous GPU-accelerators. To address the problem,
this paper proposes a distributed training system with hybrid parallelism
support on heterogeneous GPU-accelerators for large-scale models. It introduces
a distributed unified communicator to realize the communication between
heterogeneous GPU-accelerators, a distributed performance predictor, and an
automatic hybrid parallel module to develop and train models efficiently with
heterogeneous GPU-accelerators. Compared to the distributed training system
with homogeneous GPU-accelerators, our system can support six different
combinations of heterogeneous GPU-accelerators and the optimal performance of
heterogeneous GPU-accelerators has achieved at least 90% of the theoretical
upper bound performance of homogeneous GPU-accelerators.
