  Centralised training with decentralised execution (CT-DE) serves as the
foundation of many leading multi-agent reinforcement learning (MARL)
algorithms. Despite its popularity, it suffers from a critical drawback due to
its reliance on learning from a single sample of the joint-action at a given
state. As agents explore and update their policies during training, these
single samples may poorly represent the actual joint-policy of the system of
agents leading to high variance gradient estimates that hinder learning. To
address this problem, we propose an enhancement tool that accommodates any
actor-critic MARL method. Our framework, Performance Enhancing Reinforcement
Learning Apparatus (PERLA), introduces a sampling technique of the agents'
joint-policy into the critics while the agents train. This leads to TD updates
that closely approximate the true expected value under the current joint-policy
rather than estimates from a single sample of the joint-action at a given
state. This produces low variance and precise estimates of expected returns,
minimising the variance in the critic estimators which typically hinders
learning. Moreover, as we demonstrate, by eliminating much of the critic
variance from the single sampling of the joint policy, PERLA enables CT-DE
methods to scale more efficiently with the number of agents. Theoretically, we
prove that PERLA reduces variance in value estimates similar to that of
decentralised training while maintaining the benefits of centralised training.
Empirically, we demonstrate PERLA's superior performance and ability to reduce
estimator variance in a range of benchmarks including Multi-agent Mujoco, and
StarCraft II Multi-agent Challenge.
