  Distributed machine learning (DML) technology makes it possible to train
large neural networks in a reasonable amount of time. Meanwhile, as the
computing power grows much faster than network capacity, network communication
has gradually become the bottleneck of DML. Current multi-tenant GPU clusters
face network contention caused by hash-collision problem which not only further
increases the overhead of communication, but also creates unfairness and
affects the user experience. In this paper, we firstly analyse how network
contention affects the training time in a cluster with 32 NVIDIA V100 GPUs.
Then we propose vClos to eliminate network contention by jointly optimizing
network topology and communication pattern in distributed training. An
OCS-vClos which introduces a layer of optical circuit switches (OCSs) in the
leaf-spine network is also proposed to reduce potential network resource
fragmentation caused by resource allocation strategy in vClos. Testbed
experiments and real-trace-based large-scale simulations are conducted to
demonstrate the superiority of vClos over existing network resource scheduling
strategies.
