  The paper considers independent reinforcement learning (IRL) for multi-agent
decision-making process in the paradigm of federated learning (FL). We show
that FL can clearly improve the policy performance of IRL in terms of training
efficiency and stability. However, since the policy parameters are trained
locally and aggregated iteratively through a central server in FL, frequent
information exchange incurs a large amount of communication overheads. To reach
a good balance between improving the model's convergence performance and
reducing the required communication and computation overheads, this paper
proposes a system utility function and develops a consensus-based optimization
scheme on top of the periodic averaging method, which introduces the consensus
algorithm into FL for the exchange of a model's local gradients. This paper
also provides novel convergence guarantees for the developed method, and
demonstrates its superior effectiveness and efficiency in improving the system
utility value through theoretical analyses and numerical simulation results.
