  The dynamic nature of resource allocation and runtime conditions on Cloud can
result in high variability in a job's runtime across multiple iterations,
leading to a poor experience. Identifying the sources of such variation and
being able to predict and adjust for them is crucial to cloud service providers
to design reliable data processing pipelines, provision and allocate resources,
adjust pricing services, meet SLOs and debug performance hazards. In this
paper, we analyze the runtime variation of millions of production SCOPE jobs on
Cosmos, an exabyte-scale internal analytics platform at Microsoft. We propose
an innovative 2-step approach to predict job runtime distribution by
characterizing typical distribution shapes combined with a classification model
with an average accuracy of >96%, out-performing traditional regression models
and better capturing long tails. We examine factors such as job plan
characteristics and inputs, resource allocation, physical cluster heterogeneity
and utilization, and scheduling policies.
  To the best of our knowledge, this is the first study on predicting
categories of runtime distributions for enterprise analytics workloads at
scale. Furthermore, we examine how our methods can be used to analyze what-if
scenarios, focusing on the impact of resource allocation, scheduling, and
physical cluster provisioning decisions on a job's runtime consistency and
predictability.
