  Multi-Agent Reinforcement Learning (MARL) is an increasingly important
research field that can model and control multiple large-scale autonomous
systems. Despite its achievements, existing multi-agent learning methods
typically involve expensive computations in terms of training time and power
arising from large observation-action space and a huge number of training
steps. Therefore, a key challenge is understanding and characterizing the
computationally intensive functions in several popular classes of MARL
algorithms during their training phases. Our preliminary experiments reveal new
insights into the key modules of MARL algorithms that limit the adoption of
MARL in real-world systems. We explore neighbor sampling strategy to improve
cache locality and observe performance improvement ranging from 26.66% (3
agents) to 27.39% (12 agents) during the computationally intensive mini-batch
sampling phase. Additionally, we demonstrate that improving the locality leads
to an end-to-end training time reduction of 10.2% (for 12 agents) compared to
existing multi-agent algorithms without significant degradation in the mean
reward.
