  Deep reinforcement learning approaches are becoming appealing for the design
of nonlinear controllers for voltage control problems, but the lack of
stability guarantees hinders their deployment in real-world scenarios. This
paper constructs a decentralized RL-based controller featuring two components:
a transient control policy and a steady-state performance optimizer. The
transient policy is parameterized as a neural network, and the steady-state
optimizer represents the gradient of the long-term operating cost function. The
two parts are synthesized through a safe gradient flow framework, which
prevents the violation of reactive power capacity constraints. We prove that if
the output of the transient controller is bounded and monotonically decreasing
with respect to its input, then the closed-loop system is asymptotically stable
and converges to the optimal steady-state solution. We demonstrate the
effectiveness of our method by conducting experiments with IEEE 13-bus and
123-bus distribution system test feeders.
