  Visual-Semantic Embedding (VSE) is a prevalent approach in image-text
retrieval by learning a joint embedding space between the image and language
modalities where semantic similarities would be preserved. The triplet loss
with hard-negative mining has become the de-facto objective for most VSE
methods. Inspired by recent progress in deep metric learning (DML) in the image
domain which gives rise to new loss functions that outperform triplet loss, in
this paper, we revisit the problem of finding better objectives for VSE in
image-text matching. Despite some attempts in designing losses based on
gradient movement, most DML losses are defined empirically in the embedding
space. Instead of directly applying these loss functions which may lead to
sub-optimal gradient updates in model parameters, in this paper we present a
novel Gradient-based Objective AnaLysis framework, or \textit{GOAL}, to
systematically analyze the combinations and reweighting of the gradients in
existing DML functions. With the help of this analysis framework, we further
propose a new family of objectives in the gradient space exploring different
gradient combinations. In the event that the gradients are not integrable to a
valid loss function, we implement our proposed objectives such that they would
directly operate in the gradient space instead of on the losses in the
embedding space. Comprehensive experiments have demonstrated that our novel
objectives have consistently improved performance over baselines across
different visual/text features and model frameworks. We also showed the
generalizability of the GOAL framework by extending it to other models using
triplet family losses including vision-language model with heavy cross-modal
interactions and have achieved state-of-the-art results on the image-text
retrieval tasks on COCO and Flick30K.
