  Proportional-integral-derivative (PID) controllers have been widely used in
the process industry. However, the satisfactory control performance of a PID
controller depends strongly on the tuning parameters. Conventional PID tuning
methods require extensive knowledge of the system model, which is not always
known especially in the case of complex dynamical systems. In contrast,
reinforcement learning-based PID tuning has gained popularity since it can
treat PID tuning as a black-box problem and deliver the optimal PID parameters
without requiring explicit process models. In this paper, we present a novel
entropy-maximizing twin-delayed deep deterministic policy gradient (EMTD3)
method for automating the PID tuning. In the proposed method, an
entropy-maximizing stochastic actor is employed at the beginning to encourage
the exploration of the action space. Then a deterministic actor is deployed to
focus on local exploitation and discover the optimal solution. The
incorporation of the entropy-maximizing term can significantly improve the
sample efficiency and assist in fast convergence to the global solution. Our
proposed method is applied to the PID tuning of a second-order system to verify
its effectiveness in improving the sample efficiency and discovering the
optimal PID parameters compared to traditional TD3.
