  This work uses the entropy-regularised relaxed stochastic control perspective
as a principled framework for designing reinforcement learning (RL) algorithms.
Herein agent interacts with the environment by generating noisy controls
distributed according to the optimal relaxed policy. The noisy policies on the
one hand, explore the space and hence facilitate learning but, on the other
hand, introduce bias by assigning a positive probability to non-optimal
actions. This exploration-exploitation trade-off is determined by the strength
of entropy regularisation. We study algorithms resulting from two entropy
regularisation formulations: the exploratory control approach, where entropy is
added to the cost objective, and the proximal policy update approach, where
entropy penalises policy divergence between consecutive episodes. We focus on
the finite horizon continuous-time linear-quadratic (LQ) RL problem, where a
linear dynamics with unknown drift coefficients is controlled subject to
quadratic costs. In this setting, both algorithms yield a Gaussian relaxed
policy. We quantify the precise difference between the value functions of a
Gaussian policy and its noisy evaluation and show that the execution noise must
be independent across time. By tuning the frequency of sampling from relaxed
policies and the parameter governing the strength of entropy regularisation, we
prove that the regret, for both learning algorithms, is of the order
$\mathcal{O}(\sqrt{N}) $ (up to a logarithmic factor) over $N$ episodes,
matching the best known result from the literature.
