  We introduce an approach for deep reinforcement learning (RL) that improves
upon the efficiency, generalization capacity, and interpretability of
conventional approaches through structured perception and relational reasoning.
It uses self-attention to iteratively reason about the relations between
entities in a scene and to guide a model-free policy. Our results show that in
a novel navigation and planning task called Box-World, our agent finds
interpretable solutions that improve upon baselines in terms of sample
complexity, ability to generalize to more complex scenes than experienced
during training, and overall performance. In the StarCraft II Learning
Environment, our agent achieves state-of-the-art performance on six mini-games
-- surpassing human grandmaster performance on four. By considering
architectural inductive biases, our work opens new directions for overcoming
important, but stubborn, challenges in deep RL.
