  Vision-Language Pre-training (VLP) models have achieved state-of-the-art
performance in numerous cross-modal tasks. Since they are optimized to capture
the statistical properties of intra- and inter-modality, there remains risk to
learn social biases presented in the data as well. In this work, we (1)
introduce a counterfactual-based bias measurement \emph{CounterBias} to
quantify the social bias in VLP models by comparing the [MASK]ed prediction
probabilities of factual and counterfactual samples; (2) construct a novel
VL-Bias dataset including 24K image-text pairs for measuring gender bias in VLP
models, from which we observed that significant gender bias is prevalent in VLP
models; and (3) propose a VLP debiasing method \emph{FairVLP} to minimize the
difference in the [MASK]ed prediction probabilities between factual and
counterfactual image-text pairs for VLP debiasing. Although CounterBias and
FairVLP focus on social bias, they are generalizable to serve as tools and
provide new insights to probe and regularize more knowledge in VLP models.
