  In certain complex optimization tasks, it becomes necessary to use multiple
measures to characterize the performance of different algorithms. This paper
presents a method that combines ordinal effect sizes with Pareto dominance to
analyze such cases. Since the method is ordinal, it can also generalize across
different optimization tasks even when the performance measurements are
differently scaled. Through a case study, we show that this method can discover
and quantify relations that would be difficult to deduce using a conventional
measure-by-measure analysis. This case study applies the method to the
evolution of robot controller repertoires using the MAP-Elites algorithm. Here,
we analyze the search performance across a large set of parametrizations;
varying mutation size and operator type, as well as map resolution, across four
different robot morphologies. We show that the average magnitude of mutations
has a bigger effect on outcomes than their precise distributions.
