  Large-scale cloud data centers have gained popularity due to their high
availability, rapid elasticity, scalability, and low cost. However, current
data centers continue to have high failure rates due to the lack of proper
resource utilization and early failure detection. To maximize resource
efficiency and reduce failure rates in large-scale cloud data centers, it is
crucial to understand the workload and failure characteristics. In this paper,
we perform a deep analysis of the 2019 Google Cluster Trace Dataset, which
contains 2.4TiB of workload traces from eight different clusters around the
world. We explore the characteristics of failed and killed jobs in Google's
production cloud and attempt to correlate them with key attributes such as
resource usage, job priority, scheduling class, job duration, and the number of
task resubmissions. Our analysis reveals several important characteristics of
failed jobs that contribute to job failure and hence, could be used for
developing an early failure prediction system. Also, we present a novel usage
analysis to identify heterogeneity in jobs and tasks submitted by users. We are
able to identify specific users who control more than half of all collection
events on a single cluster. We contend that these characteristics could be
useful in developing an early job failure prediction system that could be
utilized for dynamic rescheduling of the job scheduler and thus improving
resource utilization in large-scale cloud data centers while reducing failure
rates.
