  Accommodating long-running deep learning (DL) training and inference jobs is
challenging on GPU clusters that use traditional batch schedulers, such as
Slurm. Given fixed wall clock time limits, DL researchers usually need to run a
sequence of batch jobs and experience long interruptions on overloaded
machines. Such interruptions significantly lower the research productivity and
QoS for services that are deployed in production. To mitigate the issues from
interruption, we investigate a set of statistical learning and reinforcement
learning (RL) techniques, including random forest, xgboost, Deep Q-Network, and
policy gradient to design a proactive provisioner using production job traces
from three GPU clusters. We follow the standard machine learning practice by
partitioning each job trace into training and validation subsets, then train
each model using the training subset and evaluate the generality using the
validation subset. We introduce Mirage, a Slurm-compatible resource provisioner
that integrates the candidate RL methods. Our experiments show that the Mirage
can reduce the interruption by 17-100% and safeguard 23%-76% of jobs with zero
interruption across varying load levels on the three clusters.
