  Reinforcement learning (RL) is a general framework that allows systems to
learn autonomously through trial-and-error interaction with their environment.
In recent years combining RL with expressive, high-capacity neural network
models has led to impressive performance in a diverse range of domains.
However, dealing with the large state and action spaces often required for
problems in the real world still remains a significant challenge. In this paper
we introduce a new simulation environment, "Gambit", designed as a tool to
build scenarios that can drive RL research in a direction useful for military
analysis. Using this environment we focus on an abstracted and simplified room
clearance scenario, where a team of blue agents have to make their way through
a building and ensure that all rooms are cleared of (and remain clear) of enemy
red agents. We implement a multi-agent version of feudal hierarchical RL that
introduces a command hierarchy where a commander at the higher level sends
orders to multiple agents at the lower level who simply have to learn to follow
these orders. We find that breaking the task down in this way allows us to
solve a number of non-trivial floorplans that require the coordination of
multiple agents much more efficiently than the standard baseline RL algorithms
we compare with. We then go on to explore how qualitatively different behaviour
can emerge depending on what we prioritise in the agent's reward function (e.g.
clearing the building quickly vs. prioritising rescuing civilians).
