  Solving temporally-extended tasks is a challenge for most reinforcement
learning (RL) algorithms [arXiv:1906.07343]. We investigate the ability of an
RL agent to learn to ask natural language questions as a tool to understand its
environment and achieve greater generalisation performance in novel,
temporally-extended environments. We do this by endowing this agent with the
ability of asking "yes-no" questions to an all-knowing Oracle. This allows the
agent to obtain guidance regarding the task at hand, while limiting the access
to new information. To study the emergence of such natural language questions
in the context of temporally-extended tasks we first train our agent in a
Mini-Grid environment. We then transfer the trained agent to a different,
harder environment. We observe a significant increase in generalisation
performance compared to a baseline agent unable to ask questions. Through
grounding its understanding of natural language in its environment, the agent
can reason about the dynamics of its environment to the point that it can ask
new, relevant questions when deployed in a novel environment.
