  Fueled by advances in distributed deep learning (DDL), recent years have
witnessed a rapidly growing demand for resource-intensive distributed/parallel
computing to process DDL computing jobs. To resolve network communication
bottleneck and load balancing issues in distributed computing, the so-called
``ring-all-reduce'' decentralized architecture has been increasingly adopted to
remove the need for dedicated parameter servers. To date, however, there
remains a lack of theoretical understanding on how to design resource
optimization algorithms for efficiently scheduling ring-all-reduce DDL jobs in
computing clusters. This motivates us to fill this gap by proposing a series of
new resource scheduling designs for ring-all-reduce DDL jobs. Our contributions
in this paper are three-fold: i) We propose a new resource scheduling
analytical model for ring-all-reduce deep learning, which covers a wide range
of objectives in DDL performance optimization (e.g., excessive training
avoidance, energy efficiency, fairness); ii) Based on the proposed performance
analytical model, we develop an efficient resource scheduling algorithm called
GADGET (greedy ring-all-reduce distributed graph embedding technique), which
enjoys a provable strong performance guarantee; iii) We conduct extensive
trace-driven experiments to demonstrate the effectiveness of the GADGET
approach and its superiority over the state of the art.
