  Conditional inference on joint textual and visual clues is a multi-modal
reasoning task that textual clues provide prior permutation or external
knowledge, which are complementary with visual content and pivotal to deducing
the correct option. Previous methods utilizing pretrained vision-language
models (VLMs) have achieved impressive performances, yet they show a lack of
multimodal context reasoning capability, especially for text-modal information.
To address this issue, we propose a Multi-modal Context Reasoning approach,
named ModCR. Compared to VLMs performing reasoning via cross modal semantic
alignment, it regards the given textual abstract semantic and objective image
information as the pre-context information and embeds them into the language
model to perform context reasoning. Different from recent vision-aided language
models used in natural language processing, ModCR incorporates the multi-view
semantic alignment information between language and vision by introducing the
learnable alignment prefix between image and text in the pretrained language
model. This makes the language model well-suitable for such multi-modal
reasoning scenario on joint textual and visual clues. We conduct extensive
experiments on two corresponding data sets and experimental results show
significantly improved performance (exact gain by 4.8% on PMR test set)
compared to previous strong baselines. Code Link:
\url{https://github.com/YunxinLi/Multimodal-Context-Reasoning}.
