  Many recent breakthroughs in multi-agent reinforcement learning (MARL)
require the use of deep neural networks, which are challenging for human
experts to interpret and understand. On the other hand, existing work on
interpretable reinforcement learning (RL) has shown promise in extracting more
interpretable decision tree-based policies from neural networks, but only in
the single-agent setting. To fill this gap, we propose the first set of
algorithms that extract interpretable decision-tree policies from neural
networks trained with MARL. The first algorithm, IVIPER, extends VIPER, a
recent method for single-agent interpretable RL, to the multi-agent setting. We
demonstrate that IVIPER learns high-quality decision-tree policies for each
agent. To better capture coordination between agents, we propose a novel
centralized decision-tree training algorithm, MAVIPER. MAVIPER jointly grows
the trees of each agent by predicting the behavior of the other agents using
their anticipated trees, and uses resampling to focus on states that are
critical for its interactions with other agents. We show that both algorithms
generally outperform the baselines and that MAVIPER-trained agents achieve
better-coordinated performance than IVIPER-trained agents on three different
multi-agent particle-world environments.
