  In this paper, we introduce a new framework to address the problem of voltage
regulation in unbalanced distribution grids with deep photovoltaic penetration.
In this framework, both real and reactive power setpoints are explicitly
controlled at each solar panel smart inverter, and the objective is to
simultaneously minimize system-wide voltage deviation and maximize solar power
output. We formulate the problem as a Markov decision process with continuous
action spaces and use proximal policy optimization, a reinforcement
learning-based approach, to solve it, without the need for any forecast or
explicit knowledge of network topology or line parameters. By representing the
system in a quasi-steady state manner, and by carefully formulating the Markov
decision process, we reduce the complexity of the problem and allow for fully
decentralized (communication-free) policies, all of which make the trained
policies much more practical and interpretable. Numerical simulations on a
240-node unbalanced distribution grid, based on a real network in Midwest U.S.,
are used to validate the proposed framework and reinforcement learning
approach.
