  Image captioning models are known to perpetuate and amplify harmful societal
bias in the training set. In this work, we aim to mitigate such gender bias in
image captioning models. While prior work has addressed this problem by forcing
models to focus on people to reduce gender misclassification, it conversely
generates gender-stereotypical words at the expense of predicting the correct
gender. From this observation, we hypothesize that there are two types of
gender bias affecting image captioning models: 1) bias that exploits context to
predict gender, and 2) bias in the probability of generating certain (often
stereotypical) words because of gender. To mitigate both types of gender
biases, we propose a framework, called LIBRA, that learns from synthetically
biased samples to decrease both types of biases, correcting gender
misclassification and changing gender-stereotypical words to more neutral ones.
Code is available at https://github.com/rebnej/LIBRA.
