  The prevalence of the Internet of things (IoT) and smart meters devices in
smart grids is providing key support for measuring and analyzing the power
consumption patterns. This approach enables end-user to play the role of
prosumers in the market and subsequently contributes to diminish the carbon
footprint and the burden on utility grids. The coordination of trading
surpluses of energy that is generated by house renewable energy resources
(RERs) and the supply of shortages by external networks (main grid) is a
necessity. This paper proposes a hierarchical architecture to manage energy in
multiple smart buildings leveraging federated deep reinforcement learning
(FDRL) with dynamic load in a distributed manner. Within the context of the
developed FDRL-based framework, each agent that is hosted in local building
energy management systems (BEMS) trains a local deep reinforcement learning
(DRL) model and shares its experience in the form of model hyperparameters to
the federation layer in the energy management system (EMS). Simulation studies
are conducted using one EMS and up to twenty smart houses that are equipped
with photovoltaic (PV) systems and batteries. This iterative training approach
enables the proposed discretized soft actor-critic (SAC) agents to aggregate
the collected knowledge to expedite the overall learning procedure and reduce
costs and CO2 emissions, while the federation approach can mitigate privacy
breaches. The numerical results confirm the performance of the proposed
framework under different daytime periods, loads, and temperatures.
