  While Multi-modal Language Models (MLMs) demonstrate impressive multimodal
ability, they still struggle on providing factual and precise responses for
tasks like visual question answering (VQA). In this paper, we address this
challenge from the perspective of contextual information. We propose Causal
Context Generation, Causal-CoG, which is a prompting strategy that engages
contextual information to enhance precise VQA during inference. Specifically,
we prompt MLMs to generate contexts, i.e, text description of an image, and
engage the generated contexts for question answering. Moreover, we investigate
the advantage of contexts on VQA from a causality perspective, introducing
causality filtering to select samples for which contextual information is
helpful. To show the effectiveness of Causal-CoG, we run extensive experiments
on 10 multimodal benchmarks and show consistent improvements, e.g., +6.30% on
POPE, +13.69% on Vizwiz and +6.43% on VQAv2 compared to direct decoding,
surpassing existing methods. We hope Casual-CoG inspires explorations of
context knowledge in multimodal models, and serves as a plug-and-play strategy
for MLM decoding.
