  Specialized accelerators such as GPUs, TPUs, FPGAs, and custom ASICs have
been increasingly deployed to train deep learning models. These accelerators
exhibit heterogeneous performance behavior across model architectures. Existing
schedulers for clusters of accelerators, which are used to arbitrate these
expensive training resources across many users, have shown how to optimize for
various multi-job, multi-user objectives, like fairness and makespan.
Unfortunately, existing schedulers largely do not consider performance
heterogeneity. In this paper, we propose Gavel, a heterogeneity-aware scheduler
that systematically generalizes a wide range of existing scheduling policies.
Gavel expresses these policies as optimization problems, making it easy to
optimize for objectives in a heterogeneity-aware way, while also being
cognizant of performance optimizations like space sharing. Gavel then uses a
round-based scheduling mechanism to ensure jobs receive their ideal allocation
given the target scheduling policy. Gavel's heterogeneity-aware policies allow
a heterogeneous cluster to sustain higher input load, and improve end
objectives such as average job completion time and makespan by up to 3.5x
compared to heterogeneity-agnostic policies.
