  Credit assignment is a core problem that distinguishes agents' marginal
contributions for optimizing cooperative strategies in multi-agent
reinforcement learning (MARL). Current credit assignment methods usually assume
synchronous decision-making among agents. However, a prerequisite for many
realistic cooperative tasks is asynchronous decision-making by agents, without
waiting for others to avoid disastrous consequences. To address this issue, we
propose an asynchronous credit assignment framework with a problem model called
ADEX-POMDP and a multiplicative value decomposition (MVD) algorithm. ADEX-POMDP
is an asynchronous problem model with extra virtual agents for a decentralized
partially observable markov decision process. We prove that ADEX-POMDP
preserves both the task equilibrium and the algorithm convergence. MVD utilizes
multiplicative interaction to efficiently capture the interactions of
asynchronous decisions, and we theoretically demonstrate its advantages in
handling asynchronous tasks. Experimental results show that on two asynchronous
decision-making benchmarks, Overcooked and POAC, MVD not only consistently
outperforms state-of-the-art MARL methods but also provides the
interpretability for asynchronous cooperation.
