  Multi-agent robotic systems are increasingly operating in real-world
environments in close proximity to humans, yet are largely controlled by policy
models with inscrutable deep neural network representations. We introduce a
method for incorporating interpretable concepts from a domain expert into
models trained through multi-agent reinforcement learning, by requiring the
model to first predict such concepts then utilize them for decision making.
This allows an expert to both reason about the resulting concept policy models
in terms of these high-level concepts at run-time, as well as intervene and
correct mispredictions to improve performance. We show that this yields
improved interpretability and training stability, with benefits to policy
performance and sample efficiency in a simulated and real-world
cooperative-competitive multi-agent game.
