  A reliable critic is central to on-policy actor-critic learning. But it
becomes challenging to learn a reliable critic in a multi-agent sparse reward
scenario due to two factors: 1) The joint action space grows exponentially with
the number of agents 2) This, combined with the reward sparseness and
environment noise, leads to large sample requirements for accurate learning. We
show that regularising the critic with spectral normalization (SN) enables it
to learn more robustly, even in multi-agent on-policy sparse reward scenarios.
Our experiments show that the regularised critic is quickly able to learn from
the sparse rewarding experience in the complex SMAC and RWARE domains. These
findings highlight the importance of regularisation in the critic for stable
learning.
