  Image repurposing is a commonly used method for spreading misinformation on
social media and online forums, which involves publishing untampered images
with modified metadata to create rumors and further propaganda. While manual
verification is possible, given vast amounts of verified knowledge available on
the internet, the increasing prevalence and ease of this form of semantic
manipulation call for the development of robust automatic ways of assessing the
semantic integrity of multimedia data. In this paper, we present a novel method
for image repurposing detection that is based on the real-world adversarial
interplay between a bad actor who repurposes images with counterfeit metadata
and a watchdog who verifies the semantic consistency between images and their
accompanying metadata, where both players have access to a reference dataset of
verified content, which they can use to achieve their goals. The proposed
method exhibits state-of-the-art performance on location-identity,
subject-identity and painting-artist verification, showing its efficacy across
a diverse set of scenarios.
