  Recent advances in quantum technology have led to the development and
manufacturing of experimental programmable quantum annealing optimizers that
contain hundreds of quantum bits. These optimizers, named `D-Wave' chips,
promise to solve practical optimization problems potentially faster than
conventional `classical' computers. Attempts to quantify the quantum nature of
these chips have been met with both excitement and skepticism but have also
brought up numerous fundamental questions pertaining to the distinguishability
of quantum annealers from their classical thermal counterparts. Here, we
propose a general method aimed at answering these, and apply it to
experimentally study the D-Wave chip. Inspired by spin-glass theory, we
generate optimization problems with a wide spectrum of `classical hardness',
which we also define. By investigating the chip's response to classical
hardness, we surprisingly find that the chip's performance scales unfavorably
as compared to several analogous classical algorithms. We detect, quantify and
discuss purely classical effects that possibly mask the quantum behavior of the
chip.
