  Deep Reinforcement Learning (RL) has considerably advanced over the past
decade. At the same time, state-of-the-art RL algorithms require a large
computational budget in terms of training time to converge. Recent work has
started to approach this problem through the lens of quantum computing, which
promises theoretical speed-ups for several traditionally hard tasks. In this
work, we examine a class of hybrid quantum-classical RL algorithms that we
collectively refer to as variational quantum deep Q-networks (VQ-DQN). We show
that VQ-DQN approaches are subject to instabilities that cause the learned
policy to diverge, study the extent to which this afflicts reproduciblity of
established results based on classical simulation, and perform systematic
experiments to identify potential explanations for the observed instabilities.
Additionally, and in contrast to most existing work on quantum reinforcement
learning, we execute RL algorithms on an actual quantum processing unit (an IBM
Quantum Device) and investigate differences in behaviour between simulated and
physical quantum systems that suffer from implementation deficiencies. Our
experiments show that, contrary to opposite claims in the literature, it cannot
be conclusively decided if known quantum approaches, even if simulated without
physical imperfections, can provide an advantage as compared to classical
approaches. Finally, we provide a robust, universal and well-tested
implementation of VQ-DQN as a reproducible testbed for future experiments.
