  Free energy-based reinforcement learning (FERL) with clamped quantum
Boltzmann machines (QBM) was shown to significantly improve the learning
efficiency compared to classical Q-learning with the restriction, however, to
discrete state-action space environments. In this paper, the FERL approach is
extended to multi-dimensional continuous state-action space environments to
open the doors for a broader range of real-world applications. First, free
energy-based Q-learning is studied for discrete action spaces, but continuous
state spaces and the impact of experience replay on sample efficiency is
assessed. In a second step, a hybrid actor-critic scheme for continuous
state-action spaces is developed based on the Deep Deterministic Policy
Gradient algorithm combining a classical actor network with a QBM-based critic.
The results obtained with quantum annealing, both simulated and with D-Wave
quantum annealing hardware, are discussed, and the performance is compared to
classical reinforcement learning methods. The environments used throughout
represent existing particle accelerator beam lines at the European Organisation
for Nuclear Research (CERN). Among others, the hybrid actor-critic agent is
evaluated on the actual electron beam line of the Advanced Plasma Wakefield
Experiment (AWAKE).
