  Many organizations routinely analyze large datasets using systems for
distributed data-parallel processing and clusters of commodity resources. Yet,
users need to configure adequate resources for their data processing jobs. This
requires significant insights into expected job runtimes and scaling behavior,
resource characteristics, input data distributions, and other factors. Unable
to estimate performance accurately, users frequently overprovision resources
for their jobs, leading to low resource utilization and high costs. In this
paper, we present major building blocks towards a collaborative approach for
optimization of data processing cluster configurations based on runtime data
and performance models. We believe that runtime data can be shared and used for
performance models across different execution contexts, significantly reducing
the reliance on the recurrence of individual processing jobs or, else,
dedicated job profiling. For this, we describe how the similarity of processing
jobs and cluster infrastructures can be employed to combine suitable data
points from local and global job executions into accurate performance models.
Furthermore, we outline approaches to performance prediction via more
context-aware and reusable models. Finally, we lay out how metrics from
previous executions can be combined with runtime monitoring to effectively
re-configure models and clusters dynamically.
