  Dialogue participants may have varying levels of knowledge about the topic
under discussion. In such cases, it is essential for speakers to adapt their
utterances by taking their audience into account. Yet, it is an open question
how such adaptation can be modelled in computational agents. In this paper, we
model a visually grounded referential game between a knowledgeable speaker and
a listener with more limited visual and linguistic experience. Inspired by
psycholinguistic theories, we endow our speaker with the ability to adapt its
referring expressions via a simulation module that monitors the effectiveness
of planned utterances from the listener's perspective. We propose an adaptation
mechanism building on plug-and-play approaches to controlled language
generation, where utterance generation is steered on the fly by the simulator
without finetuning the speaker's underlying language model. Our results and
analyses show that our approach is effective: the speaker's utterances become
closer to the listener's domain of expertise, which leads to higher
communicative success.
