  Multimodal AI models capable of associating images and text hold promise for
numerous domains, ranging from automated image captioning to accessibility
applications for blind and low-vision users. However, uncertainty about bias
has in some cases limited their adoption and availability. In the present work,
we study 43 CLIP vision-language models to determine whether they learn
human-like facial impression biases, and we find evidence that such biases are
reflected across three distinct CLIP model families. We show for the first time
that the the degree to which a bias is shared across a society predicts the
degree to which it is reflected in a CLIP model. Human-like impressions of
visually unobservable attributes, like trustworthiness and sexuality, emerge
only in models trained on the largest dataset, indicating that a better fit to
uncurated cultural data results in the reproduction of increasingly subtle
social biases. Moreover, we use a hierarchical clustering approach to show that
dataset size predicts the extent to which the underlying structure of facial
impression bias resembles that of facial impression bias in humans. Finally, we
show that Stable Diffusion models employing CLIP as a text encoder learn facial
impression biases, and that these biases intersect with racial biases in Stable
Diffusion XL-Turbo. While pretrained CLIP models may prove useful for
scientific studies of bias, they will also require significant dataset curation
when intended for use as general-purpose models in a zero-shot setting.
