  Children can rapidly generalize compositionally-constructed rules to unseen
test sets. On the other hand, deep reinforcement learning (RL) agents need to
be trained over millions of episodes, and their ability to generalize to unseen
combinations remains unclear. Hence, we investigate the compositional abilities
of RL agents, using the task of navigating to specified color-shape targets in
synthetic 3D environments. First, we show that when RL agents are naively
trained to navigate to target color-shape combinations, they implicitly learn
to decompose the combinations, allowing them to (re-)compose these and succeed
at held-out test combinations ("compositional learning"). Second, when agents
are pretrained to learn invariant shape and color concepts ("concept
learning"), the number of episodes subsequently needed for compositional
learning decreased by 20 times. Furthermore, only agents trained on both
concept and compositional learning could solve a more complex,
out-of-distribution environment in zero-shot fashion. Finally, we verified that
only text encoders pretrained on image-text datasets (e.g. CLIP) reduced the
number of training episodes needed for our agents to demonstrate compositional
learning, and also generalized to 5 unseen colors in zero-shot fashion.
Overall, our results are the first to demonstrate that RL agents can be trained
to implicitly learn concepts and compositionality, to solve more complex
environments in zero-shot fashion.
