  Selecting appropriate computational resources for data processing jobs on
large clusters is difficult, even for expert users like data engineers.
Inadequate choices can result in vastly increased costs, without significantly
improving performance. One crucial aspect of selecting an efficient resource
configuration is avoiding memory bottlenecks. By knowing the required memory of
a job in advance, the search space for an optimal resource configuration can be
greatly reduced.
  Therefore, we present Ruya, a method for memory-aware optimization of data
processing cluster configurations based on iteratively exploring a
narrowed-down search space. First, we perform job profiling runs with small
samples of the dataset on just a single machine to model the job's memory usage
patterns. Second, we prioritize cluster configurations with a suitable amount
of total memory and within this reduced search space, we iteratively search for
the best cluster configuration with Bayesian optimization. This search process
stops once it converges on a configuration that is believed to be optimal for
the given job. In our evaluation on a dataset with 1031 Spark and Hadoop jobs,
we see a reduction of search iterations to find an optimal configuration by
around half, compared to the baseline.
