  Recent work has shown that large text-based neural language models, trained
with conventional supervised learning objectives, acquire a surprising
propensity for few- and one-shot learning. Here, we show that an embodied agent
situated in a simulated 3D world, and endowed with a novel dual-coding external
memory, can exhibit similar one-shot word learning when trained with
conventional reinforcement learning algorithms. After a single introduction to
a novel object via continuous visual perception and a language prompt ("This is
a dax"), the agent can re-identify the object and manipulate it as instructed
("Put the dax on the bed"). In doing so, it seamlessly integrates short-term,
within-episode knowledge of the appropriate referent for the word "dax" with
long-term lexical and motor knowledge acquired across episodes (i.e. "bed" and
"putting"). We find that, under certain training conditions and with a
particular memory writing mechanism, the agent's one-shot word-object binding
generalizes to novel exemplars within the same ShapeNet category, and is
effective in settings with unfamiliar numbers of objects. We further show how
dual-coding memory can be exploited as a signal for intrinsic motivation,
stimulating the agent to seek names for objects that may be useful for later
executing instructions. Together, the results demonstrate that deep neural
networks can exploit meta-learning, episodic memory and an explicitly
multi-modal environment to account for 'fast-mapping', a fundamental pillar of
human cognitive development and a potentially transformative capacity for
agents that interact with human users.
