  Generative multimodal models based on diffusion models have seen tremendous
growth and advances in recent years. Models such as DALL-E and Stable Diffusion
have become increasingly popular and successful at creating images from texts,
often combining abstract ideas. However, like other deep learning models, they
also reflect social biases they inherit from their training data, which is
often crawled from the internet. Manually auditing models for biases can be
very time and resource consuming and is further complicated by the unbounded
and unconstrained nature of inputs these models can take. Research into bias
measurement and quantification has generally focused on small single-stage
models working on a single modality. Thus the emergence of multistage
multimodal models requires a different approach. In this paper, we propose
Multimodal Composite Association Score (MCAS) as a new method of measuring
gender bias in multimodal generative models. Evaluating both DALL-E 2 and
Stable Diffusion using this approach uncovered the presence of gendered
associations of concepts embedded within the models. We propose MCAS as an
accessible and scalable method of quantifying potential bias for models with
different modalities and a range of potential biases.
