  This paper presents a novel federated reinforcement learning (Fed-RL)
methodology to enhance the cyber resiliency of networked microgrids. We
formulate a resilient reinforcement learning (RL) training setup which (a)
generates episodic trajectories injecting adversarial actions at primary
control reference signals of the grid forming (GFM) inverters and (b) trains
the RL agents (or controllers) to alleviate the impact of the injected
adversaries. To circumvent data-sharing issues and concerns for proprietary
privacy in multi-party-owned networked grids, we bring in the aspects of
federated machine learning and propose a novel Fed-RL algorithm to train the RL
agents. To this end, the conventional horizontal Fed-RL approaches using
decoupled independent environments fail to capture the coupled dynamics in a
networked microgrid, which leads us to propose a multi-agent vertically
federated variation of actor-critic algorithms, namely federated soft
actor-critic (FedSAC) algorithm. We created a customized simulation setup
encapsulating microgrid dynamics in the GridLAB-D/HELICS co-simulation platform
compatible with the OpenAI Gym interface for training RL agents. Finally, the
proposed methodology is validated with numerical examples of modified IEEE
123-bus benchmark test systems consisting of three coupled microgrids.
