  This paper describes how domain knowledge of power system operators can be
integrated into reinforcement learning (RL) frameworks to effectively learn
agents that control the grid's topology to prevent thermal cascading. Typical
RL-based topology controllers fail to perform well due to the large
search/optimization space. Here, we propose an actor-critic-based agent to
address the problem's combinatorial nature and train the agent using the RL
environment developed by RTE, the French TSO. To address the challenge of the
large optimization space, a curriculum-based approach with reward tuning is
incorporated into the training procedure by modifying the environment using
network physics for enhanced agent learning. Further, a parallel training
approach on multiple scenarios is employed to avoid biasing the agent to a few
scenarios and make it robust to the natural variability in grid operations.
Without these modifications to the training procedure, the RL agent failed for
most test scenarios, illustrating the importance of properly integrating domain
knowledge of physical systems for real-world RL learning. The agent was tested
by RTE for the 2019 learning to run the power network challenge and was awarded
the 2nd place in accuracy and 1st place in speed. The developed code is
open-sourced for public use.
