  Text-to-image models take a sentence (i.e., prompt) and generate images
associated with this input prompt. These models have created award wining-art,
videos, and even synthetic datasets. However, text-to-image (T2I) models can
generate images that underrepresent minorities based on race and sex. This
paper investigates which word in the input prompt is responsible for bias in
generated images. We introduce a method for computing scores for each word in
the prompt; these scores represent its influence on biases in the model's
output. Our method follows the principle of \emph{explaining by removing},
leveraging masked language models to calculate the influence scores. We perform
experiments on Stable Diffusion to demonstrate that our method identifies the
replication of societal stereotypes in generated images.
