  We propose a novel discriminative model that learns embeddings from
multilingual and multi-modal data, meaning that our model can take advantage of
images and descriptions in multiple languages to improve embedding quality. To
that end, we introduce a modification of a pairwise contrastive estimation
optimisation function as our training objective. We evaluate our embeddings on
an image-sentence ranking (ISR), a semantic textual similarity (STS), and a
neural machine translation (NMT) task. We find that the additional multilingual
signals lead to improvements on both the ISR and STS tasks, and the
discriminative cost can also be used in re-ranking $n$-best lists produced by
NMT models, yielding strong improvements.
