  Recent breakthroughs in self supervised training have led to a new class of
pretrained vision language models. While there have been investigations of bias
in multimodal models, they have mostly focused on gender and racial bias,
giving much less attention to other relevant groups, such as minorities with
regard to religion, nationality, sexual orientation, or disabilities. This is
mainly due to lack of suitable benchmarks for such groups. We seek to address
this gap by providing a visual and textual bias benchmark called MMBias,
consisting of around 3,800 images and phrases covering 14 population subgroups.
We utilize this dataset to assess bias in several prominent self supervised
multimodal models, including CLIP, ALBEF, and ViLT. Our results show that these
models demonstrate meaningful bias favoring certain groups. Finally, we
introduce a debiasing method designed specifically for such large pre-trained
models that can be applied as a post-processing step to mitigate bias, while
preserving the remaining accuracy of the model.
