  The increasing proliferation of misinformation and its alarming impact have
motivated both industry and academia to develop approaches for misinformation
detection and fact checking. Recent advances on large language models (LLMs)
have shown remarkable performance in various tasks, but whether and how LLMs
could help with misinformation detection remains relatively underexplored. Most
of existing state-of-the-art approaches either do not consider evidence and
solely focus on claim related features or assume the evidence to be provided.
Few approaches consider evidence retrieval as part of the misinformation
detection but rely on fine-tuning models. In this paper, we investigate the
potential of LLMs for misinformation detection in a zero-shot setting. We
incorporate an evidence retrieval component into the process as it is crucial
to gather pertinent information from various sources to detect the veracity of
claims. To this end, we propose a novel re-ranking approach for multimodal
evidence retrieval using both LLMs and large vision-language models (LVLM). The
retrieved evidence samples (images and texts) serve as the input for an
LVLM-based approach for multimodal fact verification (LVLM4FV). To enable a
fair evaluation, we address the issue of incomplete ground truth for evidence
samples in an existing evidence retrieval dataset by annotating a more complete
set of evidence samples for both image and text retrieval. Our experimental
results on two datasets demonstrate the superiority of the proposed approach in
both evidence retrieval and fact verification tasks and also better
generalization capability across dataset compared to the supervised baseline.
