  In this paper we describe a novel framework and algorithms for discovering
image patch patterns from a large corpus of weakly supervised image-caption
pairs generated from news events. Current pattern mining techniques attempt to
find patterns that are representative and discriminative, we stipulate that our
discovered patterns must also be recognizable by humans and preferably with
meaningful names. We propose a new multimodal pattern mining approach that
leverages the descriptive captions often accompanying news images to learn
semantically meaningful image patch patterns. The mutltimodal patterns are then
named using words mined from the associated image captions for each pattern. A
novel evaluation framework is provided that demonstrates our patterns are 26.2%
more semantically meaningful than those discovered by the state of the art
vision only pipeline, and that we can provide tags for the discovered images
patches with 54.5% accuracy with no direct supervision. Our methods also
discover named patterns beyond those covered by the existing image datasets
like ImageNet. To the best of our knowledge this is the first algorithm
developed to automatically mine image patch patterns that have strong semantic
meaning specific to high-level news events, and then evaluate these patterns
based on that criteria.
