  Benchmarking is how the performance of a computing system is determined.
Surprisingly, even for classical computers this is not a straightforward
process. One must choose the appropriate benchmark and metrics to extract
meaningful results. Different benchmarks test the system in different ways and
each individual metric may or may not be of interest. Choosing the appropriate
approach is tricky. The situation is even more open ended for quantum
computers, where there is a wider range of hardware, fewer established
guidelines, and additional complicating factors. Notably, quantum noise
significantly impacts performance and is difficult to model accurately. Here,
we discuss benchmarking of quantum computers from a computer architecture
perspective and provide numerical simulations highlighting challenges which
suggest caution.
