  Effective communication requires adapting to the idiosyncrasies of each
communicative context--such as the common ground shared with each partner.
Humans demonstrate this ability to specialize to their audience in many
contexts, such as the popular game Dixit. We take inspiration from Dixit to
formulate a multi-agent image reference game where a (trained) speaker model is
rewarded for describing a target image such that one (pretrained) listener
model can correctly identify it among distractors, but another listener cannot.
To adapt, the speaker must exploit differences in the knowledge it shares with
the different listeners. We show that finetuning an attention-based adapter
between a CLIP vision encoder and a large language model in this contrastive,
multi-agent setting gives rise to context-dependent natural language
specialization from rewards only, without direct supervision. Through
controlled experiments, we show that training a speaker with two listeners that
perceive differently, using our method, allows the speaker to adapt to the
idiosyncracies of the listeners. Furthermore, we show zero-shot transfer of the
specialization to real-world data. Our experiments demonstrate a method for
specializing grounded language models without direct supervision and highlight
the interesting research challenges posed by complex multi-agent communication.
