  MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that
extends the popular single-agent method, DDPG, to multi-agent scenarios.
Importantly, DDPG is an algorithm designed for continuous action spaces, where
the gradient of the state-action value function exists. For this algorithm to
work in discrete action spaces, discrete gradient estimation must be performed.
For MADDPG, the Gumbel-Softmax (GS) estimator is used -- a reparameterisation
which relaxes a discrete distribution into a similar continuous one. This
method, however, is statistically biased, and a recent MARL benchmarking paper
suggests that this bias makes MADDPG perform poorly in grid-world situations,
where the action space is discrete. Fortunately, many alternatives to the GS
exist, boasting a wide range of properties. This paper explores several of
these alternatives and integrates them into MADDPG for discrete grid-world
scenarios. The corresponding impact on various performance metrics is then
measured and analysed. It is found that one of the proposed estimators performs
significantly better than the original GS in several tasks, achieving up to 55%
higher returns, along with faster convergence.
