  Job submissions of parallel applications to production supercomputer systems
will have to be carefully tuned in terms of the job submission parameters to
obtain minimum response times. In this work, we have developed an end-to-end
resource management framework that uses predictions of queue waiting and
execution times to minimize response times of user jobs submitted to
supercomputer systems. Our method for predicting queue waiting times adaptively
chooses a prediction method based on the cluster structure of similar jobs. Our
strategy for execution time predictions dynamically learns the impact of load
on execution times and uses this to predict a set of execution time ranges for
the target job. We have developed two resource management techniques that
employ these predictions, one that selects the number of processors for
execution and the other that also dynamically changes the job submission time.
Using workload simulations of large supercomputer traces, we show large-scale
improvements in predictions and reductions in response times over existing
techniques and baseline strategies.
