  As a new research area, quantum software testing lacks systematic testing
benchmarks to assess testing techniques' effectiveness. Recently, some
open-source benchmarks and mutation analysis tools have emerged. However, there
is insufficient evidence on how various quantum circuit characteristics (e.g.,
circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate
Optimization Algorithm), and mutation characteristics (e.g., mutation
operators) affect the detection of mutants in quantum circuits. Studying such
relations is important to systematically design faulty benchmarks with varied
attributes (e.g., the difficulty in detecting a seeded fault) to facilitate
assessing the cost-effectiveness of quantum software testing techniques
efficiently. To this end, we present a large-scale empirical evaluation with
more than 700K faulty benchmarks (quantum circuits) generated by mutating 382
real-world quantum circuits. Based on the results, we provide valuable insights
for researchers to define systematic quantum mutation analysis techniques. We
also provide a tool to recommend mutants to users based on chosen
characteristics (e.g., a quantum algorithm type) and the required difficulty of
detecting mutants. Finally, we also provide faulty benchmarks that can already
be used to assess the cost-effectiveness of quantum software testing
techniques.
