  Computer vision often treats human perception as homogeneous: an implicit
assumption that visual stimuli are perceived similarly by everyone. This
assumption is reflected in the way researchers collect datasets and train
vision models. By contrast, literature in cross-cultural psychology and
linguistics has provided evidence that people from different cultural
backgrounds observe vastly different concepts even when viewing the same visual
stimuli. In this paper, we study how these differences manifest themselves in
vision-language datasets and models, using language as a proxy for culture. By
comparing textual descriptions generated across 7 languages for the same
images, we find significant differences in the semantic content and linguistic
expression. When datasets are multilingual as opposed to monolingual,
descriptions have higher semantic coverage on average, where coverage is
measured using scene graphs, model embeddings, and linguistic taxonomies. For
example, multilingual descriptions have on average 29.9% more objects, 24.5%
more relations, and 46.0% more attributes than a set of monolingual captions.
When prompted to describe images in different languages, popular models (e.g.
LLaVA) inherit this bias and describe different parts of the image. Moreover,
finetuning models on captions from one language performs best on corresponding
test data from that language, while finetuning on multilingual data performs
consistently well across all test data compositions. Our work points towards
the need to account for and embrace the diversity of human perception in the
computer vision community.
