  The ability to accurately estimate job runtime properties allows a scheduler
to effectively schedule jobs. State-of-the-art online cluster job schedulers
use history-based learning, which uses past job execution information to
estimate the runtime properties of newly arrived jobs. However, with fast-paced
development in cluster technology (in both hardware and software) and changing
user inputs, job runtime properties can change over time, which lead to
inaccurate predictions. In this paper, we explore the potential and limitation
of real-time learning of job runtime properties, by proactively sampling and
scheduling a small fraction of the tasks of each job. Such a
task-sampling-based approach exploits the similarity among runtime properties
of the tasks of the same job and is inherently immune to changing job behavior.
Our study focuses on two key questions in comparing task-sampling-based
learning (learning in space) and history-based learning (learning in time): (1)
Can learning in space be more accurate than learning in time? (2) If so, can
delaying scheduling the remaining tasks of a job till the completion of sampled
tasks be more than compensated by the improved accuracy and result in improved
job performance? Our analytical and experimental analysis of 3 production
traces with different skew and job distribution shows that learning in space
can be substantially more accurate. Our simulation and testbed evaluation on
Azure of the two learning approaches anchored in a generic job scheduler using
3 production cluster job traces shows that despite its online overhead,
learning in space reduces the average Job Completion Time (JCT) by 1.28x,
1.56x, and 1.32x compared to the prior-art history-based predictor.
