  In cooperative multi-agent reinforcement learning (MARL), how to design a
suitable reward signal to accelerate learning and stabilize convergence is a
critical problem. The global reward signal assigns the same global reward to
all agents without distinguishing their contributions, while the local reward
signal provides different local rewards to each agent based solely on
individual behavior. Both of the two reward assignment approaches have some
shortcomings: the former might encourage lazy agents, while the latter might
produce selfish agents.
  In this paper, we study reward design problem in cooperative MARL based on
packet routing environments. Firstly, we show that the above two reward signals
are prone to produce suboptimal policies. Then, inspired by some observations
and considerations, we design some mixed reward signals, which are
off-the-shelf to learn better policies. Finally, we turn the mixed reward
signals into the adaptive counterparts, which achieve best results in our
experiments. Other reward signals are also discussed in this paper. As reward
design is a very fundamental problem in RL and especially in MARL, we hope that
MARL researchers can rethink the rewards used in their systems.
