  DNN-based cross-modal retrieval is a research hotspot to retrieve across
different modalities as image and text, but existing methods often face the
challenge of insufficient cross-modal training data. In single-modal scenario,
similar problem is usually relieved by transferring knowledge from large-scale
auxiliary datasets (as ImageNet). Knowledge from such single-modal datasets is
also very useful for cross-modal retrieval, which can provide rich general
semantic information that can be shared across different modalities. However,
it is challenging to transfer useful knowledge from single-modal (as image)
source domain to cross-modal (as image/text) target domain. Knowledge in source
domain cannot be directly transferred to both two different modalities in
target domain, and the inherent cross-modal correlation contained in target
domain provides key hints for cross-modal retrieval which should be preserved
during transfer process. This paper proposes Cross-modal Hybrid Transfer
Network (CHTN) with two subnetworks: Modal-sharing transfer subnetwork utilizes
the modality in both source and target domains as a bridge, for transferring
knowledge to both two modalities simultaneously; Layer-sharing correlation
subnetwork preserves the inherent cross-modal semantic correlation to further
adapt to cross-modal retrieval task. Cross-modal data can be converted to
common representation by CHTN for retrieval, and comprehensive experiment on 3
datasets shows its effectiveness.
