  Despite numerous successes, the field of reinforcement learning (RL) remains
far from matching the impressive generalisation power of human behaviour
learning. One possible way to help bridge this gap be to provide RL agents with
richer, more human-like feedback expressed in natural language. To investigate
this idea, we first extend BabyAI to automatically generate language feedback
from the environment dynamics and goal condition success. Then, we modify the
Decision Transformer architecture to take advantage of this additional signal.
We find that training with language feedback either in place of or in addition
to the return-to-go or goal descriptions improves agents' generalisation
performance, and that agents can benefit from feedback even when this is only
available during training, but not at inference.
