  Quantum Reinforcement Learning (QRL) emerged as a branch of reinforcement
learning (RL) that uses quantum submodules in the architecture of the
algorithm. One branch of QRL focuses on the replacement of neural networks (NN)
by variational quantum circuits (VQC) as function approximators. Initial works
have shown promising results on classical environments with discrete action
spaces, but many of the proposed architectural design choices of the VQC lack a
detailed investigation. Hence, in this work we investigate the impact of VQC
design choices such as angle embedding, encoding block architecture and
postprocessesing on the training capabilities of QRL agents. We show that VQC
design greatly influences training performance and heuristically derive
enhancements for the analyzed components. Additionally, we show how to design a
QRL agent in order to solve classical environments with continuous action
spaces and benchmark our agents against classical feed-forward NNs.
