  Hadoop has become the de facto standard for processing large data in today's
cloud environment. The performance of Hadoop in the cloud has a direct impact
on many important applications ranging from web analytic, web indexing, image
and document processing to high-performance scientific computing. However,
because of the scale, complexity and dynamic nature of the cloud, failures are
common and these failures often impact the performance of jobs running in
Hadoop. Although Hadoop possesses built-in failure detection and recovery
mechanisms, several scheduled jobs still fail because of unforeseen events in
the cloud environment. A single task failure can cause the failure of the whole
job and unpredictable job running times. In this report, we propose ATLAS
(AdapTive faiLure-Aware Scheduler), a new scheduler for Hadoop that can adapt
its scheduling decisions to events occurring in the cloud environment. Using
statistical models, ATLAS predicts task failures and adjusts its scheduling
decisions on the fly to reduce task failure occurrences. We implement ATLAS in
the Hadoop framework of Amazon Elastic MapReduce (EMR) and perform a case study
to compare its performance with those of the FIFO, Fair and Capacity
schedulers. Results show that ATLAS can reduce the percentage of failed jobs by
up to 28% and the percentage of failed tasks by up to 39%, and the total
execution time of jobs by 10 minutes on average. ATLAS also reduces CPU and
memory usages.
