  Multi-agent reinforcement learning (MARL) tasks often utilize a centralized
training with decentralized execution (CTDE) framework. QMIX is a successful
CTDE method that learns a credit assignment function to derive local value
functions from a global value function, defining a deterministic local policy.
However, QMIX is hindered by its poor exploration strategy. While maximum
entropy reinforcement learning (RL) promotes better exploration through
stochastic policies, QMIX's process of credit assignment conflicts with the
maximum entropy objective and the decentralized execution requirement, making
it unsuitable for maximum entropy RL. In this paper, we propose an enhancement
to QMIX by incorporating an additional local Q-value learning method within the
maximum entropy RL framework. Our approach constrains the local Q-value
estimates to maintain the correct ordering of all actions. Due to the
monotonicity of the QMIX value function, these updates ensure that locally
optimal actions align with globally optimal actions. We theoretically prove the
monotonic improvement and convergence of our method to an optimal solution.
Experimentally, we validate our algorithm in matrix games, Multi-Agent Particle
Environment and demonstrate state-of-the-art performance in SMAC-v2.
