  Deep reinforcement learning has been applied successfully to solve various
real-world problems and the number of its applications in the multi-agent
settings has been increasing. Multi-agent learning distinctly poses significant
challenges in the effort to allocate a concealed communication medium. Agents
receive thorough knowledge from the medium to determine subsequent actions in a
distributed nature. Apparently, the goal is to leverage the cooperation of
multiple agents to achieve a designated objective efficiently. Recent studies
typically combine a specialized neural network with reinforcement learning to
enable communication between agents. This approach, however, limits the number
of agents or necessitates the homogeneity of the system. In this paper, we have
proposed a more scalable approach that not only deals with a great number of
agents but also enables collaboration between dissimilar functional agents and
compatibly combined with any deep reinforcement learning methods. Specifically,
we create a global communication map to represent the status of each agent in
the system visually. The visual map and the environmental state are fed to a
shared-parameter network to train multiple agents concurrently. Finally, we
select the Asynchronous Advantage Actor-Critic (A3C) algorithm to demonstrate
our proposed scheme, namely Visual communication map for Multi-agent A3C
(VMA3C). Simulation results show that the use of visual communication map
improves the performance of A3C regarding learning speed, reward achievement,
and robustness in multi-agent problems.
