  Existing image-text modality alignment in Vision Language Models (VLMs)
treats each text token equally in an autoregressive manner. Despite being
simple and effective, this method results in sub-optimal cross-modal alignment
by over-emphasizing the text tokens that are less correlated with or even
contradictory with the input images. In this paper, we advocate for assigning
distinct contributions for each text token based on its visual correlation.
Specifically, we present by contrasting image inputs, the difference in
prediction logits on each text token provides strong guidance of visual
correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet
effective re-weighting strategy that prioritizes training visually correlated
tokens. Our experimental results demonstrate that CAL consistently improves
different types of VLMs across different resolutions and model sizes on various
benchmark datasets. Importantly, our method incurs minimal additional
computational overhead, rendering it highly efficient compared to alternative
data scaling strategies. Codes are available at
https://github.com/foundation-multimodal-models/CAL.
