  Using only image-sentence pairs, weakly-supervised visual-textual grounding
aims to learn region-phrase correspondences of the respective entity mentions.
Compared to the supervised approach, learning is more difficult since bounding
boxes and textual phrases correspondences are unavailable. In light of this, we
propose the Semantic Prior Refinement Model (SPRM), whose predictions are
obtained by combining the output of two main modules. The first untrained
module aims to return a rough alignment between textual phrases and bounding
boxes. The second trained module is composed of two sub-components that refine
the rough alignment to improve the accuracy of the final phrase-bounding box
alignments. The model is trained to maximize the multimodal similarity between
an image and a sentence, while minimizing the multimodal similarity of the same
sentence and a new unrelated image, carefully selected to help the most during
training. Our approach shows state-of-the-art results on two popular datasets,
Flickr30k Entities and ReferIt, shining especially on ReferIt with a 9.6%
absolute improvement. Moreover, thanks to the untrained component, it reaches
competitive performances just using a small fraction of training examples.
