  Models for Visual Question Answering (VQA) often rely on the spurious
correlations, i.e., the language priors, that appear in the biased samples of
training set, which make them brittle against the out-of-distribution (OOD)
test data. Recent methods have achieved promising progress in overcoming this
problem by reducing the impact of biased samples on model training. However,
these models reveal a trade-off that the improvements on OOD data severely
sacrifice the performance on the in-distribution (ID) data (which is dominated
by the biased samples). Therefore, we propose a novel contrastive learning
approach, MMBS, for building robust VQA models by Making the Most of Biased
Samples. Specifically, we construct positive samples for contrastive learning
by eliminating the information related to spurious correlation from the
original training samples and explore several strategies to use the constructed
positive samples for training. Instead of undermining the importance of biased
samples in model training, our approach precisely exploits the biased samples
for unbiased information that contributes to reasoning. The proposed method is
compatible with various VQA backbones. We validate our contributions by
achieving competitive performance on the OOD dataset VQA-CP v2 while preserving
robust performance on the ID dataset VQA v2.
