  Reference-model-assisted deep reinforcement learning (DRL) for inverter-based
Volt-Var Control (IB-VVC) in active distribution networks is proposed. We
investigate that a large action space increases the learning difficulties of
DRL and degrades the optimization performance in the process of generating data
and training neural networks. To reduce the action space of DRL, we design a
reference-model-assisted DRL approach. We introduce definitions of the
reference model, reference-model-based optimization, and reference actions. The
reference-model-assisted DRL learns the residual actions between the reference
actions and optimal actions, rather than learning the optimal actions directly.
Since the residual actions are considerably smaller than the optimal actions
for a reference model, we can design a smaller action space for the
reference-model-assisted DRL. It reduces the learning difficulties of DRL and
optimises the performance of the reference-model-assisted DRL approach. It is
noteworthy that the reference-model-assisted DRL approach is compatible with
any policy gradient DRL algorithms for continuous action problems. This work
takes the soft actor-critic algorithm as an example and designs a
reference-model-assisted soft actor-critic algorithm. Simulations show that 1)
large action space degrades the performance of DRL in the whole training stage,
and 2) reference-model-assisted DRL requires fewer iteration times and returns
a better optimization performance.
