  Humans extract useful abstractions of the world from noisy sensory data.
Serial reproduction allows us to study how people construe the world through a
paradigm similar to the game of telephone, where one person observes a stimulus
and reproduces it for the next to form a chain of reproductions. Past serial
reproduction experiments typically employ a single sensory modality, but humans
often communicate abstractions of the world to each other through language. To
investigate the effect language on the formation of abstractions, we implement
a novel multimodal serial reproduction framework by asking people who receive a
visual stimulus to reproduce it in a linguistic format, and vice versa. We ran
unimodal and multimodal chains with both humans and GPT-4 and find that adding
language as a modality has a larger effect on human reproductions than GPT-4's.
This suggests human visual and linguistic representations are more dissociable
than those of GPT-4.
