  Experimental comparisons of performance represent an important aspect of
research on optimization algorithms. In this work we present a methodology for
defining the required sample sizes for designing experiments with desired
statistical properties for the comparison of two methods on a given problem
class. The proposed approach allows the experimenter to define desired levels
of accuracy for estimates of mean performance differences on individual problem
instances, as well as the desired statistical power for comparing mean
performances over a problem class of interest. The method calculates the
required number of problem instances, and runs the algorithms on each test
instance so that the accuracy of the estimated differences in performance is
controlled at the predefined level. Two examples illustrate the application of
the proposed method, and its ability to achieve the desired statistical
properties with a methodologically sound definition of the relevant sample
sizes.
