  We introduce Segment-Phrase Table (SPT), a large collection of bijective
associations between textual phrases and their corresponding segmentations.
Leveraging recent progress in object recognition and natural language
semantics, we show how we can successfully build a high-quality segment-phrase
table using minimal human supervision. More importantly, we demonstrate the
unique value unleashed by this rich bimodal resource, for both vision as well
as natural language understanding. First, we show that fine-grained textual
labels facilitate contextual reasoning that helps in satisfying semantic
constraints across image segments. This feature enables us to achieve
state-of-the-art segmentation results on benchmark datasets. Next, we show that
the association of high-quality segmentations to textual phrases aids in richer
semantic understanding and reasoning of these textual phrases. Leveraging this
feature, we motivate the problem of visual entailment and visual paraphrasing,
and demonstrate its utility on a large dataset.
