  Although High Performance Computing (HPC) users understand basic resource
requirements such as the number of CPUs and memory limits, internal
infrastructural utilization data is exclusively leveraged by cluster operators,
who use it to configure batch schedulers. This task is challenging and
increasingly complex due to ever larger cluster scales and heterogeneity of
modern scientific workflows. As a result, HPC systems achieve low utilization
with long job completion times (makespans). To tackle these challenges, we
propose a co-scheduling algorithm based on an adaptive reinforcement learning
algorithm, where application profiling is combined with cluster monitoring. The
resulting cluster scheduler matches resource utilization to application
performance in a fine-grained manner (i.e., operating system level). As opposed
to nominal allocations, we apply decision trees to model applications' actual
resource usage, which are used to estimate how much resource capacity from one
allocation can be co-allocated to additional applications. Our algorithm learns
from incorrect co-scheduling decisions and adapts from changing environment
conditions, and evaluates when such changes cause resource contention that
impacts quality of service metrics such as jobs slowdowns. We integrate our
algorithm in an HPC resource manager that combines Slurm and Mesos for job
scheduling and co-allocation, respectively. Our experimental evaluation
performed in a dedicated cluster executing a mix of four real different
scientific workflows demonstrates improvements on cluster utilization of up to
51% even in high load scenarios, with 55% average queue makespan reductions
under low loads.
