  Next Generation (NextG) networks are expected to support demanding tactile
internet applications such as augmented reality and connected autonomous
vehicles. Whereas recent innovations bring the promise of larger link capacity,
their sensitivity to the environment and erratic performance defy traditional
model-based control rationales. Zero-touch data-driven approaches can improve
the ability of the network to adapt to the current operating conditions. Tools
such as reinforcement learning (RL) algorithms can build optimal control policy
solely based on a history of observations. Specifically, deep RL (DRL), which
uses a deep neural network (DNN) as a predictor, has been shown to achieve good
performance even in complex environments and with high dimensional inputs.
However, the training of DRL models require a large amount of data, which may
limit its adaptability to ever-evolving statistics of the underlying
environment. Moreover, wireless networks are inherently distributed systems,
where centralized DRL approaches would require excessive data exchange, while
fully distributed approaches may result in slower convergence rates and
performance degradation. In this paper, to address these challenges, we propose
a federated learning (FL) approach to DRL, which we refer to federated DRL
(F-DRL), where base stations (BS) collaboratively train the embedded DNN by
only sharing models' weights rather than training data. We evaluate two
distinct versions of F-DRL, value and policy based, and show the superior
performance they achieve compared to distributed and centralized DRL.
