  Images with visual and scene text content are ubiquitous in everyday life.
However, current image interpretation systems are mostly limited to using only
the visual features, neglecting to leverage the scene text content. In this
paper, we propose to jointly use scene text and visual channels for robust
semantic interpretation of images. We do not only extract and encode visual and
scene text cues, but also model their interplay to generate a contextual joint
embedding with richer semantics. The contextual embedding thus generated is
applied to retrieval and classification tasks on multimedia images, with scene
text content, to demonstrate its effectiveness. In the retrieval framework, we
augment our learned text-visual semantic representation with scene text cues,
to mitigate vocabulary misses that may have occurred during the semantic
embedding. To deal with irrelevant or erroneous recognition of scene text, we
also apply query-based attention to our text channel. We show how the
multi-channel approach, involving visual semantics and scene text, improves
upon state of the art.
