  Maximum entropy reinforcement learning (RL) methods have been successfully
applied to a range of challenging sequential decision-making and control tasks.
However, most of existing techniques are designed for discrete-time systems. As
a first step toward their extension to continuous-time systems, this paper
considers continuous-time deterministic optimal control problems with entropy
regularization. Applying the dynamic programming principle, we derive a novel
class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal
value function of the maximum entropy control problem corresponds to the unique
viscosity solution of the HJB equation. Our maximum entropy formulation is
shown to enhance the regularity of the viscosity solution and to be
asymptotically consistent as the effect of entropy regularization diminishes. A
salient feature of the HJB equations is computational tractability. Generalized
Hopf-Lax formulas can be used to solve the HJB equations in a tractable
grid-free manner without the need for numerically optimizing the Hamiltonian.
We further show that the optimal control is uniquely characterized as Gaussian
in the case of control affine systems and that, for linear-quadratic problems,
the HJB equation is reduced to a Riccati equation, which can be used to obtain
an explicit expression of the optimal control. Lastly, we discuss how to extend
our results to continuous-time model-free RL by taking an adaptive dynamic
programming approach. To our knowledge, the resulting algorithms are the first
data-driven control methods that use an information theoretic exploration
mechanism in continuous time.
