  Researchers have integrated exploration techniques into multi-agent
reinforcement learning (MARL) algorithms, drawing on their remarkable success
in deep reinforcement learning. Nonetheless, exploration in MARL presents a
more substantial challenge, as agents need to coordinate their efforts in order
to achieve comprehensive state coverage. Reaching a unanimous agreement on
which kinds of states warrant exploring can be a struggle for agents in this
context. We introduce \textbf{M}ulti-agent \textbf{E}xploration based on
\textbf{S}ub-state \textbf{E}ntropy (MESE) to address this limitation. This
novel approach incentivizes agents to explore states cooperatively by directing
them to achieve consensus via an extra team reward. Calculating the additional
reward is based on the novelty of the current sub-state that merits cooperative
exploration. MESE employs a conditioned entropy approach to select the
sub-state, using particle-based entropy estimation to calculate the entropy.
MESE is a plug-and-play module that can be seamlessly integrated into most
existing MARL algorithms, which makes it a highly effective tool for
reinforcement learning. Our experiments demonstrate that MESE can substantially
improve the MAPPO's performance on various tasks in the StarCraft multi-agent
challenge (SMAC).
