  The triplet loss with semi-hard negatives has become the de facto choice for
image-caption retrieval (ICR) methods that are optimized from scratch. Recent
progress in metric learning has given rise to new loss functions that
outperform the triplet loss on tasks such as image retrieval and representation
learning. We ask whether these findings generalize to the setting of ICR by
comparing three loss functions on two ICR methods. We answer this question
negatively: the triplet loss with semi-hard negative mining still outperforms
newly introduced loss functions from metric learning on the ICR task. To gain a
better understanding of these outcomes, we introduce an analysis method to
compare loss functions by counting how many samples contribute to the gradient
w.r.t. the query representation during optimization. We find that loss
functions that result in lower evaluation scores on the ICR task, in general,
take too many (non-informative) samples into account when computing a gradient
w.r.t. the query representation, which results in sub-optimal performance. The
triplet loss with semi-hard negatives is shown to outperform the other loss
functions, as it only takes one (hard) negative into account when computing the
gradient.
