  Online misinformation is a prevalent societal issue, with adversaries relying
on tools ranging from cheap fakes to sophisticated deep fakes. We are motivated
by the threat scenario where an image is used out of context to support a
certain narrative. While some prior datasets for detecting image-text
inconsistency generate samples via text manipulation, we propose a dataset
where both image and text are unmanipulated but mismatched. We introduce
several strategies for automatically retrieving convincing images for a given
caption, capturing cases with inconsistent entities or semantic context. Our
large-scale automatically generated NewsCLIPpings Dataset: (1) demonstrates
that machine-driven image repurposing is now a realistic threat, and (2)
provides samples that represent challenging instances of mismatch between text
and image in news that are able to mislead humans. We benchmark several
state-of-the-art multimodal models on our dataset and analyze their performance
across different pretraining domains and visual backbones.
