  In many real-world multi-robot tasks, high-quality solutions often require a
team of robots to perform asynchronous actions under decentralized control.
Decentralized multi-agent reinforcement learning methods have difficulty
learning decentralized policies because of the environment appearing to be
non-stationary due to other agents also learning at the same time. In this
paper, we address this challenge by proposing a macro-action-based
decentralized multi-agent double deep recurrent Q-net (MacDec-MADDRQN) which
trains each decentralized Q-net using a centralized Q-net for action selection.
A generalized version of MacDec-MADDRQN with two separate training
environments, called Parallel-MacDec-MADDRQN, is also presented to leverage
either centralized or decentralized exploration. The advantages and the
practical nature of our methods are demonstrated by achieving near-centralized
results in simulation and having real robots accomplish a warehouse tool
delivery task in an efficient way.
