  Cross-modal embeddings, between textual and visual modalities, aim to
organise multimodal instances by their semantic correlations. State-of-the-art
approaches use maximum-margin methods, based on the hinge-loss, to enforce a
constant margin m, to separate projections of multimodal instances from
different categories. In this paper, we propose a novel scheduled adaptive
maximum-margin (SAM) formulation that infers triplet-specific constraints
during training, therefore organising instances by adaptively enforcing
inter-category and inter-modality correlations. This is supported by a
scheduled adaptive margin function, that is smoothly activated, replacing a
static margin by an adaptively inferred one reflecting triplet-specific
semantic correlations while accounting for the incremental learning behaviour
of neural networks to enforce category cluster formation and enforcement.
Experiments on widely used datasets show that our model improved upon
state-of-the-art approaches, by achieving a relative improvement of up to
~12.5% over the second best method, thus confirming the effectiveness of our
scheduled adaptive margin formulation.
