  Text-image cross-modal retrieval is a challenging task in the field of
language and vision. Most previous approaches independently embed images and
sentences into a joint embedding space and compare their similarities. However,
previous approaches rarely explore the interactions between images and
sentences before calculating similarities in the joint space. Intuitively, when
matching between images and sentences, human beings would alternatively attend
to regions in images and words in sentences, and select the most salient
information considering the interaction between both modalities. In this paper,
we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively
controls the information flow for message passing across modalities. Our
approach not only takes comprehensive and fine-grained cross-modal interactions
into account, but also properly handles negative pairs and irrelevant
information with an adaptive gating scheme. Moreover, instead of conventional
joint embedding approaches for text-image matching, we infer the matching score
based on the fused features, and propose a hardest negative binary
cross-entropy loss for training. Results on COCO and Flickr30k significantly
surpass state-of-the-art methods, demonstrating the effectiveness of our
approach.
