  Exploration of the high-dimensional state action space is one of the biggest
challenges in Reinforcement Learning (RL), especially in multi-agent domain. We
present a novel technique called Experience Augmentation, which enables a
time-efficient and boosted learning based on a fast, fair and thorough
exploration to the environment. It can be combined with arbitrary off-policy
MARL algorithms and is applicable to either homogeneous or heterogeneous
environments. We demonstrate our approach by combining it with MADDPG and
verifing the performance in two homogeneous and one heterogeneous environments.
In the best performing scenario, the MADDPG with experience augmentation
reaches to the convergence reward of vanilla MADDPG with 1/4 realistic time,
and its convergence beats the original model by a significant margin. Our
ablation studies show that experience augmentation is a crucial ingredient
which accelerates the training process and boosts the convergence.
