  We discuss guidelines for evaluating the performance of parameterized
stochastic solvers for optimization problems, with particular attention to
systems that employ novel hardware, such as digital quantum processors running
variational algorithms, analog processors performing quantum annealing, or
coherent Ising Machines. We illustrate through an example a benchmarking
procedure grounded in the statistical analysis of the expectation of a given
performance metric measured in a test environment. In particular, we discuss
the necessity and cost of setting parameters that affect the algorithm's
performance. The optimal value of these parameters could vary significantly
between instances of the same target problem. We present an open-source
software package that facilitates the design, evaluation, and visualization of
practical parameter tuning strategies for complex use of the heterogeneous
components of the solver. We examine in detail an example using parallel
tempering and a simulator of a photonic Coherent Ising Machine computing and
display the scoring of an illustrative baseline family of parameter-setting
strategies that feature an exploration-exploitation trade-off.
