  In the near-term "NISQ"-era of noisy, intermediate-scale, quantum hardware
and beyond, reliably determining the quality of quantum devices becomes
increasingly important: users need to be able to compare them with one another,
and make an estimate whether they are capable of performing a given task ahead
of time. In this work, we develop and release an advanced quantum benchmarking
framework in order to help assess the state of the art of current quantum
devices. Our testing framework measures the performance of universal quantum
devices in a hardware-agnostic way, with metrics that are aimed to facilitate
an intuitive understanding of which device is likely to outperform others on a
given task. This is achieved through six structured tests that allow for an
immediate, visual assessment of how devices compare. Each test is designed with
scalability in mind, making this framework not only suitable for testing the
performance of present-day quantum devices, but also of those released in the
foreseeable future. The series of tests are motivated by real-life scenarios,
and therefore emphasise the interplay between various relevant characteristics
of quantum devices, such as qubit count, connectivity, and gate and measurement
fidelity. We present the benchmark results of twenty-one different quantum
devices from IBM, Rigetti and IonQ.
