  A one-step two-critic deep reinforcement learning (OSTC-DRL) approach for
inverter-based volt-var control (IB-VVC) in active distribution networks is
proposed in this paper. Firstly, considering IB-VVC can be formulated as a
single-period optimization problem, we formulate the IB-VVC as a one-step
Markov decision process rather than the standard Markov decision process, which
simplifies the DRL learning task. Then we design the one-step actor-critic DRL
scheme which is a simplified version of recent DRL algorithms, and it avoids
the issue of Q value overestimation successfully. Furthermore, considering two
objectives of VVC: minimizing power loss and eliminating voltage violation, we
utilize two critics to approximate the rewards of two objectives separately. It
simplifies the approximation tasks of each critic, and avoids the interaction
effect between two objectives in the learning process of critic. The OSTC-DRL
approach integrates the one-step actor-critic DRL scheme and the two-critic
technology. Based on the OSTC-DRL, we design two centralized DRL algorithms.
Further, we extend the OSTC-DRL to multi-agent OSTC-DRL for decentralized
IB-VVC and design two multi-agent DRL algorithms. Simulations demonstrate that
the proposed OSTC-DRL has a faster convergence rate and a better control
performance, and the multi-agent OSTC-DRL works well for decentralized IB-VVC
problems.
