  Despite the recent attention to DeepFakes, one of the most prevalent ways to
mislead audiences on social media is the use of unaltered images in a new but
false context. To address these challenges and support fact-checkers, we
propose a new method that automatically detects out-of-context image and text
pairs. Our key insight is to leverage the grounding of image with text to
distinguish out-of-context scenarios that cannot be disambiguated with language
alone. We propose a self-supervised training strategy where we only need a set
of captioned images. At train time, our method learns to selectively align
individual objects in an image with textual claims, without explicit
supervision. At test time, we check if both captions correspond to the same
object(s) in the image but are semantically different, which allows us to make
fairly accurate out-of-context predictions. Our method achieves 85%
out-of-context detection accuracy. To facilitate benchmarking of this task, we
create a large-scale dataset of 200K images with 450K textual captions from a
variety of news websites, blogs, and social media posts. The dataset and source
code is publicly available at
https://shivangi-aneja.github.io/projects/cosmos/.
