  Vision-language (VL) understanding tasks evaluate models' comprehension of
complex visual scenes through multiple-choice questions. However, we have
identified two dataset biases that models can exploit as shortcuts to resolve
various VL tasks correctly without proper understanding. The first type of
dataset bias is \emph{Unbalanced Matching} bias, where the correct answer
overlaps the question and image more than the incorrect answers. The second
type of dataset bias is \emph{Distractor Similarity} bias, where incorrect
answers are overly dissimilar to the correct answer but significantly similar
to other incorrect answers within the same sample. To address these dataset
biases, we first propose Adversarial Data Synthesis (ADS) to generate synthetic
training and debiased evaluation data. We then introduce Intra-sample
Counterfactual Training (ICT) to assist models in utilizing the synthesized
training data, particularly the counterfactual data, via focusing on
intra-sample differentiation. Extensive experiments demonstrate the
effectiveness of ADS and ICT in consistently improving model performance across
different benchmarks, even in domain-shifted scenarios.
