  Big data processing at the production scale presents a highly complex
environment for resource optimization (RO), a problem crucial for meeting
performance goals and budgetary constraints of analytical users. The RO problem
is challenging because it involves a set of decisions (the partition count,
placement of parallel instances on machines, and resource allocation to each
instance), requires multi-objective optimization (MOO), and is compounded by
the scale and complexity of big data systems while having to meet stringent
time constraints for scheduling. This paper presents a MaxCompute-based
integrated system to support multi-objective resource optimization via
fine-grained instance-level modeling and optimization. We propose a new
architecture that breaks RO into a series of simpler problems, new fine-grained
predictive models, and novel optimization methods that exploit these models to
make effective instance-level recommendations in a hierarchical MOO framework.
Evaluation using production workloads shows that our new RO system could reduce
37-72% latency and 43-78% cost at the same time, compared to the current
optimizer and scheduler, while running in 0.02-0.23s.
