  In the evaluation of quantum annealers, metrics based on ground state success
rates have two major drawbacks. First, evaluation requires computation time for
both quantum and classical processors that grows exponentially with problem
size. This makes evaluation itself computationally prohibitive. Second, results
are heavily dependent on the effects of analog noise on the quantum processors,
which is an engineering issue that complicates the study of the underlying
quantum annealing algorithm. We introduce a novel "time-to-target" metric which
avoids these two issues by challenging software solvers to match the results
obtained by a quantum annealer in a short amount of time. We evaluate D-Wave's
latest quantum annealer, the D-Wave 2X system, on an array of problem classes
and find that it performs well on several input classes relative to state of
the art software solvers running single-threaded on a CPU.
