  Federated reinforcement learning (RL) enables collaborative decision making
of multiple distributed agents without sharing local data trajectories. In this
work, we consider a multi-task setting, in which each agent has its own private
reward function corresponding to different tasks, while sharing the same
transition kernel of the environment. Focusing on infinite-horizon tabular
Markov decision processes, the goal is to learn a globally optimal policy that
maximizes the sum of the discounted total rewards of all the agents in a
decentralized manner, where each agent only communicates with its neighbors
over some prescribed graph topology. We develop federated vanilla and
entropy-regularized natural policy gradient (NPG) methods under softmax
parameterization, where gradient tracking is applied to the global Q-function
to mitigate the impact of imperfect information sharing. We establish
non-asymptotic global convergence guarantees under exact policy evaluation,
which are nearly independent of the size of the state-action space and
illuminate the impacts of network size and connectivity. To the best of our
knowledge, this is the first time that global convergence is established for
federated multi-task RL using policy optimization. Moreover, the convergence
behavior of the proposed algorithms is robust against inexactness of policy
evaluation.
