  While decentralized training is attractive in multi-agent reinforcement
learning (MARL) for its excellent scalability and robustness, its inherent
coordination challenges in collaborative tasks result in numerous interactions
for agents to learn good policies. To alleviate this problem, action advising
methods make experienced agents share their knowledge about what to do, while
less experienced agents strictly follow the received advice. However, this
method of sharing and utilizing knowledge may hinder the team's exploration of
better states, as agents can be unduly influenced by suboptimal or even adverse
advice, especially in the early stages of learning. Inspired by the fact that
humans can learn not only from the success but also from the failure of others,
this paper proposes a novel knowledge sharing framework called
Cautiously-Optimistic kNowledge Sharing (CONS). CONS enables each agent to
share both positive and negative knowledge and cautiously assimilate knowledge
from others, thereby enhancing the efficiency of early-stage exploration and
the agents' robustness to adverse advice. Moreover, considering the continuous
improvement of policies, agents value negative knowledge more in the early
stages of learning and shift their focus to positive knowledge in the later
stages. Our framework can be easily integrated into existing Q-learning based
methods without introducing additional training costs. We evaluate CONS in
several challenging multi-agent tasks and find it excels in environments where
optimal behavioral patterns are difficult to discover, surpassing the baselines
in terms of convergence rate and final performance.
