  Cross-modal retrieval has drawn wide interest for retrieval across different
modalities of data. However, existing methods based on DNN face the challenge
of insufficient cross-modal training data, which limits the training
effectiveness and easily leads to overfitting. Transfer learning is for
relieving the problem of insufficient training data, but it mainly focuses on
knowledge transfer only from large-scale datasets as single-modal source domain
to single-modal target domain. Such large-scale single-modal datasets also
contain rich modal-independent semantic knowledge that can be shared across
different modalities. Besides, large-scale cross-modal datasets are very
labor-consuming to collect and label, so it is significant to fully exploit the
knowledge in single-modal datasets for boosting cross-modal retrieval. This
paper proposes modal-adversarial hybrid transfer network (MHTN), which to the
best of our knowledge is the first work to realize knowledge transfer from
single-modal source domain to cross-modal target domain, and learn cross-modal
common representation. It is an end-to-end architecture with two subnetworks:
(1) Modal-sharing knowledge transfer subnetwork is proposed to jointly transfer
knowledge from a large-scale single-modal dataset in source domain to all
modalities in target domain with a star network structure, which distills
modal-independent supplementary knowledge for promoting cross-modal common
representation learning. (2) Modal-adversarial semantic learning subnetwork is
proposed to construct an adversarial training mechanism between common
representation generator and modality discriminator, making the common
representation discriminative for semantics but indiscriminative for modalities
to enhance cross-modal semantic consistency during transfer process.
Comprehensive experiments on 4 widely-used datasets show its effectiveness and
generality.
