  Out-of-context (OOC) misinformation poses a significant challenge in
multimodal fact-checking, where images are paired with texts that misrepresent
their original context to support false narratives. Recent research in
evidence-based OOC detection has seen a trend towards increasingly complex
architectures, incorporating Transformers, foundation models, and large
language models. In this study, we introduce a simple yet robust baseline,
which assesses MUltimodal SimilaritiEs (MUSE), specifically the similarity
between image-text pairs and external image and text evidence. Our results
demonstrate that MUSE, when used with conventional classifiers like Decision
Tree, Random Forest, and Multilayer Perceptron, can compete with and even
surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets.
Furthermore, integrating MUSE in our proposed "Attentive Intermediate
Transformer Representations" (AITR) significantly improved performance, by 3.3%
and 7.5% on NewsCLIPpings and VERITE, respectively. Nevertheless, the success
of MUSE, relying on surface-level patterns and shortcuts, without examining
factuality and logical inconsistencies, raises critical questions about how we
define the task, construct datasets, collect external evidence and overall, how
we assess progress in the field. We release our code at:
https://github.com/stevejpapad/outcontext-misinfo-progress
