  Reinforcement learning has significant applications for multi-agent systems,
especially in unknown dynamic environments. However, most multi-agent
reinforcement learning (MARL) algorithms suffer from such problems as
exponential computation complexity in the joint state-action space, which makes
it difficult to scale up to realistic multi-agent problems. In this paper, a
novel algorithm named negotiation-based MARL with sparse interactions (NegoSI)
is presented. In contrast to traditional sparse-interaction based MARL
algorithms, NegoSI adopts the equilibrium concept and makes it possible for
agents to select the non-strict Equilibrium Dominating Strategy Profile
(non-strict EDSP) or Meta equilibrium for their joint actions. The presented
NegoSI algorithm consists of four parts: the equilibrium-based framework for
sparse interactions, the negotiation for the equilibrium set, the minimum
variance method for selecting one joint action and the knowledge transfer of
local Q-values. In this integrated algorithm, three techniques, i.e., unshared
value functions, equilibrium solutions and sparse interactions are adopted to
achieve privacy protection, better coordination and lower computational
complexity, respectively. To evaluate the performance of the presented NegoSI
algorithm, two groups of experiments are carried out regarding three criteria:
steps of each episode (SEE), rewards of each episode (REE) and average runtime
(AR). The first group of experiments is conducted using six grid world games
and shows fast convergence and high scalability of the presented algorithm.
Then in the second group of experiments NegoSI is applied to an intelligent
warehouse problem and simulated results demonstrate the effectiveness of the
presented NegoSI algorithm compared with other state-of-the-art MARL
algorithms.
