  To solve difficult tasks, humans ask questions to acquire knowledge from
external sources. In contrast, classical reinforcement learning agents lack
such an ability and often resort to exploratory behavior. This is exacerbated
as few present-day environments support querying for knowledge. In order to
study how agents can be taught to query external knowledge via language, we
first introduce two new environments: the grid-world-based Q-BabyAI and the
text-based Q-TextWorld. In addition to physical interactions, an agent can
query an external knowledge source specialized for these environments to gather
information. Second, we propose the "Asking for Knowledge" (AFK) agent, which
learns to generate language commands to query for meaningful knowledge that
helps solve the tasks. AFK leverages a non-parametric memory, a pointer
mechanism and an episodic exploration bonus to tackle (1) irrelevant
information, (2) a large query language space, (3) delayed reward for making
meaningful queries. Extensive experiments demonstrate that the AFK agent
outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld
environments.
