  Intelligent robots need to achieve abstract objectives using concrete,
spatiotemporally complex sensory information and motor control. Tabula rasa
deep reinforcement learning (RL) has tackled demanding tasks in terms of either
visual, abstract, or physical reasoning, but solving these jointly remains a
formidable challenge. One recent, unsolved benchmark task that integrates these
challenges is Mujoban, where a robot needs to arrange 3D warehouses generated
from 2D Sokoban puzzles. We explore whether integrated tasks like Mujoban can
be solved by composing RL modules together in a sense-plan-act hierarchy, where
modules have well-defined roles similarly to classic robot architectures.
Unlike classic architectures that are typically model-based, we use only
model-free modules trained with RL or supervised learning. We find that our
modular RL approach dramatically outperforms the state-of-the-art monolithic RL
agent on Mujoban. Further, learned modules can be reused when, e.g., using a
different robot platform to solve the same task. Together our results give
strong evidence for the importance of research into modular RL designs. Project
website: https://sites.google.com/view/modular-rl/
