  Visual-Semantic Embedding (VSE) aims to learn an embedding space where
related visual and semantic instances are close to each other. Recent VSE
models tend to design complex structures to pool visual and semantic features
into fixed-length vectors and use hard triplet loss for optimization. However,
we find that: (1) combining simple pooling methods is no worse than these
sophisticated methods; and (2) only considering the most
difficult-to-distinguish negative sample leads to slow convergence and poor
Recall@K improvement. To this end, we propose an adaptive pooling strategy that
allows the model to learn how to aggregate features through a combination of
simple pooling methods. We also introduce a strategy to dynamically select a
group of negative samples to make the optimization converge faster and perform
better. Experimental results on Flickr30K and MS-COCO demonstrate that a
standard VSE using our pooling and optimization strategies outperforms current
state-of-the-art systems (at least 1.0% on the metrics of recall) in
image-to-text and text-to-image retrieval. Source code of our experiments is
available at https://github.com/96-Zachary/vse_2ad.
