  The paper considers independent reinforcement learning (IRL) for multi-agent
collaborative decision-making in the paradigm of federated learning (FL).
However, FL generates excessive communication overheads between agents and a
remote central server, especially when it involves a large number of agents or
iterations. Besides, due to the heterogeneity of independent learning
environments, multiple agents may undergo asynchronous Markov decision
processes (MDPs), which will affect the training samples and the model's
convergence performance. On top of the variation-aware periodic averaging (VPA)
method and the policy-based deep reinforcement learning (DRL) algorithm (i.e.,
proximal policy optimization (PPO)), this paper proposes two advanced
optimization schemes orienting to stochastic gradient descent (SGD): 1) A
decay-based scheme gradually decays the weights of a model's local gradients
with the progress of successive local updates, and 2) By representing the
agents as a graph, a consensus-based scheme studies the impact of exchanging a
model's local gradients among nearby agents from an algebraic connectivity
perspective. This paper also provides novel convergence guarantees for both
developed schemes, and demonstrates their superior effectiveness and efficiency
in improving the system's utility value through theoretical analyses and
simulation results.
