  The World Wide Web has become a popular source for gathering information and
news. Multimodal information, e.g., enriching text with photos, is typically
used to convey the news more effectively or to attract attention. Photo content
can range from decorative, depict additional important information, or can even
contain misleading information. Therefore, automatic approaches to quantify
cross-modal consistency of entity representation can support human assessors to
evaluate the overall multimodal message, for instance, with regard to bias or
sentiment. In some cases such measures could give hints to detect fake news,
which is an increasingly important topic in today's society. In this paper, we
introduce a novel task of cross-modal consistency verification in real-world
news and present a multimodal approach to quantify the entity coherence between
image and text. Named entity linking is applied to extract persons, locations,
and events from news texts. Several measures are suggested to calculate
cross-modal similarity for these entities using state of the art approaches. In
contrast to previous work, our system automatically gathers example data from
the Web and is applicable to real-world news. Results on two novel datasets
that cover different languages, topics, and domains demonstrate the feasibility
of our approach. Datasets and code are publicly available to foster research
towards this new direction.
