  Load balance is important for MapReduce to reduce job duration, increase
parallel efficiency, etc. Previous work focuses on coarse-grained scheduling.
This study concerns fine-grained scheduling on MapReduce operations. Each
operation represents one invocation of the Map or Reduce function. Scheduling
MapReduce operations is difficult due to highly screwed operation loads, no
support to collect workload statistics, and high complexity of the scheduling
problem. So current implementations adopt simple strategies, leading to poor
load balance. To address these difficulties, we design an algorithm to schedule
operations based on the key distribution of intermediate pairs. The algorithm
involves a sub-program for selecting operations for task slots, and we name it
the Balanced Subset Sum (BSS) problem. We discuss properties of BSS and design
exact and approximation algorithms for it. To transparently incorporate these
algorithms into MapReduce, we design a communication mechanism to collect
statistics, and a pipeline within Reduce tasks to increase resource
utilization. To the best of our knowledge, this is the first work on scheduling
MapReduce workload at this fine-grained level. Experiments on PUMA [T+12]
benchmarks show consistent performance improvement. The job duration can be
reduced by up to 37%, compared with standard MapReduce.
