  Vision-language models are growing in popularity and public visibility to
generate, edit, and caption images at scale; but their outputs can perpetuate
and amplify societal biases learned during pre-training on uncurated image-text
pairs from the internet. Although debiasing methods have been proposed, we
argue that these measurements of model bias lack validity due to dataset bias.
We demonstrate there are spurious correlations in COCO Captions, the most
commonly used dataset for evaluating bias, between background context and the
gender of people in-situ. This is problematic because commonly-used bias
metrics (such as Bias@K) rely on per-gender base rates. To address this issue,
we propose a novel dataset debiasing pipeline to augment the COCO dataset with
synthetic, gender-balanced contrast sets, where only the gender of the subject
is edited and the background is fixed. However, existing image editing methods
have limitations and sometimes produce low-quality images; so, we introduce a
method to automatically filter the generated images based on their similarity
to real images. Using our balanced synthetic contrast sets, we benchmark bias
in multiple CLIP-based models, demonstrating how metrics are skewed by
imbalance in the original COCO images. Our results indicate that the proposed
approach improves the validity of the evaluation, ultimately contributing to
more realistic understanding of bias in vision-language models.
