  In multi-agent reinforcement learning (MARL), the Centralized Training with
Decentralized Execution (CTDE) framework is pivotal but struggles due to a gap:
global state guidance in training versus reliance on local observations in
execution, lacking global signals. Inspired by human societal consensus
mechanisms, we introduce the Hierarchical Consensus-based Multi-Agent
Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL
employs contrastive learning to foster a global consensus among agents,
enabling cooperative behavior without direct communication. This approach
enables agents to form a global consensus from local observations, using it as
an additional piece of information to guide collaborative actions during
execution. To cater to the dynamic requirements of various tasks, consensus is
divided into multiple layers, encompassing both short-term and long-term
considerations. Short-term observations prompt the creation of an immediate,
low-layer consensus, while long-term observations contribute to the formation
of a strategic, high-layer consensus. This process is further refined through
an adaptive attention mechanism that dynamically adjusts the influence of each
consensus layer. This mechanism optimizes the balance between immediate
reactions and strategic planning, tailoring it to the specific demands of the
task at hand. Extensive experiments and real-world applications in multi-robot
systems showcase our framework's superior performance, marking significant
advancements over baselines.
