  This paper describes our participant system for the multi-modal fact
verification (Factify) challenge at AAAI 2022. Despite the recent advance in
text based verification techniques and large pre-trained multimodal models
cross vision and language, very limited work has been done in applying
multimodal techniques to automate fact checking process, particularly
considering the increasing prevalence of claims and fake news about images and
videos on social media. In our work, the challenge is treated as multimodal
entailment task and framed as multi-class classification. Two baseline
approaches are proposed and explored including an ensemble model (combining two
uni-modal models) and a multi-modal attention network (modeling the interaction
between image and text pair from claim and evidence document). We conduct
several experiments investigating and benchmarking different SoTA pre-trained
transformers and vision models in this work. Our best model is ranked first in
leaderboard which obtains a weighted average F-measure of 0.77 on both
validation and test set. Exploratory analysis of dataset is also carried out on
the Factify data set and uncovers salient patterns and issues (e.g., word
overlapping, visual entailment correlation, source bias) that motivates our
hypothesis. Finally, we highlight challenges of the task and multimodal dataset
for future research.
