  Companies build separate training and inference GPU clusters for deep
learning, and use separate schedulers to manage them. This leads to problems
for both training and inference: inference clusters have low GPU utilization
when the traffic load is low; training jobs often experience long queueing time
due to lack of resources. We introduce Aryl, a new cluster scheduler to address
these problems. Aryl introduces capacity loaning to loan idle inference GPU
servers for training jobs. It further exploits elastic scaling that scales a
training job's GPU allocation to better utilize loaned resources. Capacity
loaning and elastic scaling create new challenges to cluster management. When
the loaned servers need to be returned, we need to minimize the number of job
preemptions; when more GPUs become available, we need to allocate them to
elastic jobs and minimize the job completion time (JCT). Aryl addresses these
combinatorial problems using principled heuristics. It introduces the notion of
server preemption cost which it greedily reduces during server reclaiming. It
further relies on the JCT reduction value defined for each additional worker
for an elastic job to solve the scheduling problem as a multiple-choice
knapsack problem. Prototype implementation on a 64-GPU testbed and large-scale
simulation with 15-day traces of over 50,000 production jobs show that Aryl
brings 1.53x and 1.50x reductions in average queuing time and JCT, and improves
cluster usage by up to 26.9% over the cluster scheduler without capacity
loaning or elastic scaling.
