  Many optimal control problems require the simultaneous output of discrete and
continuous control variables. These problems are usually formulated as
mixed-integer optimal control (MIOC) problems, which are challenging to solve
due to the complexity of the solution space. Numerical methods such as
branch-and-bound are computationally expensive and undesirable for real-time
control. This paper proposes a novel hybrid-action reinforcement learning
(HARL) algorithm, twin delayed deep deterministic actor-Q (TD3AQ), for MIOC
problems. TD3AQ combines the advantages of both actor-critic and Q-learning
methods, and can handle the discrete and continuous action spaces
simultaneously. The proposed algorithm is evaluated on a plug-in hybrid
electric vehicle (PHEV) energy management problem, where real-time control of
the discrete variables, clutch engagement/disengagement and gear shift, and
continuous variable, engine torque, is essential to maximize fuel economy while
satisfying driving constraints. Simulation outcomes demonstrate that TD3AQ
achieves control results close to optimality when compared with dynamic
programming (DP), with just 4.69% difference. Furthermore, it surpasses the
performance of baseline reinforcement learning algorithms.
