  In the near-term, hybrid quantum-classical algorithms hold great potential
for outperforming classical approaches. Understanding how these two computing
paradigms work in tandem is critical for identifying areas where such hybrid
algorithms could provide a quantum advantage. In this work, we study a
QAOA-based quantum optimization algorithm by implementing the Variational
Quantum Factoring (VQF) algorithm. We execute experimental demonstrations using
a superconducting quantum processor and investigate the trade-off between
quantum resources (number of qubits and circuit depth) and the probability that
a given biprime is successfully factored. In our experiments, the integers
1099551473989, 3127, and 6557 are factored with 3, 4, and 5 qubits,
respectively, using a QAOA ansatz with up to 8 layers and we are able to
identify the optimal number of circuit layers for a given instance to maximize
success probability. Furthermore, we demonstrate the impact of different noise
sources on the performance of QAOA and reveal the coherent error caused by the
residual ZZ-coupling between qubits as a dominant source of error in the
superconducting quantum processor.
