  Pretraining on human corpus and then finetuning in a simulator has become a
standard pipeline for training a goal-oriented dialogue agent. Nevertheless, as
soon as the agents are finetuned to maximize task completion, they suffer from
the so-called language drift phenomenon: they slowly lose syntactic and
semantic properties of language as they only focus on solving the task. In this
paper, we propose a generic approach to counter language drift called Seeded
iterated learning (SIL). We periodically refine a pretrained student agent by
imitating data sampled from a newly generated teacher agent. At each time step,
the teacher is created by copying the student agent, before being finetuned to
maximize task completion. SIL does not require external syntactic constraint
nor semantic knowledge, making it a valuable task-agnostic finetuning protocol.
We evaluate SIL in a toy-setting Lewis Game, and then scale it up to the
translation game with natural language. In both settings, SIL helps counter
language drift as well as it improves the task completion compared to
baselines.
