Correlation Hashing Network for Efficient Cross-Modal Retrieval
Yue Cao
Mingsheng Long
Jianmin Wang
School of Software
Tsinghua University
Beijing
China
Tsinghua National Laboratory for Information Science and Technology
{\tt\small caoyue10@gmail.com\quad \{mingsheng
jimwang\
Due to the storage and retrieval efficiency, hashing has been widely deployed to approximate nearest neighbor search for large-scale multimedia retrieval. Cross-modal hashing, which improves the quality of hash coding by exploiting the semantic correlation across different modalities, has received increasing attention recently. For most existing cross-modal hashing methods, an object is first represented as of vector of hand-crafted or machine-learned features, followed by another separate quantization step that generates binary codes. However, suboptimal hash coding may be produced, because the quantization error is not statistically minimized and the feature representation is not optimally compatible with the binary coding. In this paper, we propose a novel Correlation Hashing Network (CHN) architecture for cross-modal hashing, in which we jointly learn good data representation tailored to hash coding and formally control the quantization error. The CHN model is a hybrid deep architecture constituting four key components: (1) an image network with multiple convolution-pooling layers to extract good image representations, and a text network with several fully-connected layers to extract good text representations; (2) a fully-connected hashing layer to generate modality-specific compact hash codes; (3) a squared cosine loss layer for capturing both cross-modal correlation and within-modal correlation; and (4) a new cosine quantization loss for controlling the quality of the binarized hash codes. Extensive experiments on standard cross-modal retrieval datasets show the proposed CHN model yields substantial boosts over latest state-of-the-art hashing methods.
CDH.tex
\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage{cvpr}
\usepackage{times}
\usepackage{amsmath,amssymb,amsthm,amsfonts,bm}
\usepackage[ruled]{algorithm2e}
\usepackage{graphicx}
\usepackage[tight,normal]{subfigure}
\usepackage{multirow,makecell}
\usepackage{url,footmisc}

%\usepackage{floatrow}
%\DeclareFloatFont{tiny}{\tiny}% "scriptsize" is defined by floatrow, "tiny" not
%\floatsetup[table]{font=tiny}

\newtheorem{definition}{Definition}
\newtheorem{problem}{Problem}
\newtheorem{lemma}{Lemma}
\newtheorem{theorem}{Theorem}

\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}

\cvprfinalcopy
\def\cvprPaperID{118} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}


% Pages are numbered in submission mode, and unnumbered in camera-ready
\ifcvprfinal\pagestyle{empty}\fi

\begin{document}

\title{Correlation Hashing Network for Efficient Cross-Modal Retrieval}

\author{
    Yue Cao, Mingsheng Long, Jianmin Wang\\
    School of Software, Tsinghua University, Beijing, China\\
    Tsinghua National Laboratory for Information Science and Technology\\
    {\tt\small caoyue10@gmail.com\quad \{mingsheng,jimwang\}@tsinghua.edu.cn}
}

\maketitle

\begin{abstract}
Due to the storage and retrieval efficiency, hashing has been widely deployed to approximate nearest neighbor search for large-scale multimedia retrieval. Cross-modal hashing, which improves the quality of hash coding by exploiting the semantic correlation across different modalities, has received increasing attention recently. For most existing cross-modal hashing methods, an object is first represented as of vector of hand-crafted or machine-learned features, followed by another separate quantization step that generates binary codes. However, suboptimal hash coding may be produced, because the quantization error is not statistically minimized and the feature representation is not optimally compatible with the binary coding. In this paper, we propose a novel Correlation Hashing Network (CHN) architecture for cross-modal hashing, in which we jointly learn good data representation tailored to hash coding and formally control the quantization error. The CHN model is a hybrid deep architecture constituting four key components: (1) an image network with multiple convolution-pooling layers to extract good image representations, and a text network with several fully-connected layers to extract good text representations; (2) a fully-connected hashing layer to generate modality-specific compact hash codes; (3) a squared cosine loss layer for capturing both cross-modal correlation and within-modal correlation; and (4) a new cosine quantization loss for controlling the quality of the binarized hash codes. Extensive experiments on standard cross-modal retrieval datasets show the proposed CHN model yields substantial boosts over latest state-of-the-art hashing methods.
\end{abstract}

\section{Introduction}\label{section:Introduction}
While multimedia big data with large volumes and high dimensions are pervasive in search engines and social networks, it has attracted increasing attention to enable approximate nearest neighbors (ANN) search across different media modalities with both computation efficiency and search quality. As relevant data from different modalities (image and text) may endow semantic correlations, it is important to support \emph{cross-modal retrieval} that returns semantically-relevant results of one modality in response to a query of different modality. An advantageous solution to the cross-modal retrieval is hashing methods \cite{cite:arxiv14hashsurvey}, which transform high-dimensional data into compact binary codes and generate similar binary codes for similar data items. In this paper, we will focus on cross-modal hashing that builds data-dependent hash coding for efficient cross-media retrieval \cite{cite:TPAMI14Wiki}. Due to large volumes and the semantic gap \cite{cite:TPAMI00SemanticGap}, effective and efficient cross-modal retrieval remains a challenge.

Many cross-modal hashing methods have been proposed to construct the correlation structures across different modalities in the process of hash function learning and indexes cross-modal data into an isomorphic Hamming space \cite{cite:CVPR10CMSSH,cite:IJCAI11CVH,cite:NIPS12CRH,cite:KDD12MLBE,cite:SIGMOD13IMH,cite:VLDB14MSAE,cite:SIGIR14DCDH,cite:CVPR14CH,cite:AAAI14SCM,cite:MM14CorrAE,cite:IJCAI15QCH}. They can be categorized into unsupervised methods \cite{cite:IJCAI11CVH,cite:NIPS12CRH,cite:SIGMOD13IMH} and supervised methods \cite{cite:CVPR10CMSSH,cite:CVPR14CH,cite:AAAI14SCM,cite:IJCAI15QCH}. While unsupervised methods are more general and can be trained without semantic labels or relevances, they are restricted by the semantic gap dilemma \cite{cite:TPAMI00SemanticGap} that high-level semantic description of an object often differs from low-level feature descriptors. Supervised methods can incorporate semantic labels or relevance feedbacks to mitigate the semantic gap \cite{cite:TPAMI00SemanticGap} and improve the hashing quality, i.e. achieve accurate search with shorter codes. 

Recently, deep hashing methods \cite{cite:AAAI14CNNH,cite:CVPR15DNNH} have shown that both feature representation and hash coding can be learned more effectively using deep neural networks \cite{cite:NIPS12CNN,cite:ICLR14NIN}, which can naturally encode nonlinear hashing functions. Other cross-modal retrieval models via deep learning \cite{cite:PAMI14CMNN,cite:JMLR14MMDL,cite:VLDB14MSAE,cite:MM14CorrAE,cite:MM14DL} have shown that deep models can capture nonlinear cross-modal correlations more effectively. These deep hashing methods have created state-of-the-art results on many benchmarks. However, a crucial disadvantage of these cross-modal deep hashing methods is that the quantization error is not statistically minimized hence the feature representation is not optimally compatible with the binary hash coding. Another potential limitation is that they generally do not adopt principled pairwise loss function to link the pairwise Hamming distances with the pairwise similarity labels and to close the gap between the Hamming distance on binary codes and the metric distance  on continuous representations. Therefore, suboptimal hash coding may be produced by existing cross-modal deep hashing methods.

In this paper, we propose Correlation Hashing Network (CHN), a new hybrid architecture for cross-modal hashing. We jointly learn good image and text representations tailored to hash coding and formally control the quantization error. The CHN model constitutes four key components: (1) an image network with multiple convolution-pooling layers to extract good image representations, and a text network with several fully-connected layers to extract good text representations; (2) a fully-connected hashing layer to generate modality-specific compact hash codes; (3) a squared cosine loss layer for capturing both cross-modal correlation and within-modal correlation; and (4) a new cosine quantization loss for controlling the quality of the binarized hash codes. Extensive experiments on standard cross-modal retrieval datasets show the proposed CHN model yields substantial boosts over latest state-of-the-art hashing methods.

\section{Related Work}\label{section:RelatedWork}
Cross-modal hashing has been an increasingly popular research topic in machine learning, computer vision, and multimedia retrieval \cite{cite:CVPR10CMSSH,cite:IJCAI11CVH,cite:NIPS12CRH,cite:KDD12MLBE,cite:SIGMOD13IMH,cite:VLDB14MSAE,cite:SIGIR14DCDH,cite:MM14CorrAE,cite:MM14IMVH,cite:AAAI14SCM,cite:CVPR14CH}. We refer readers to \cite{cite:arxiv14hashsurvey} for a comprehensive survey.

Existing cross-modal hashing methods can be roughly categorized into unsupervised methods and supervised methods. IMH \cite{cite:SIGMOD13IMH} and CVH \cite{cite:IJCAI11CVH} are unsupervised methods, which extend spectral hashing \cite{cite:NIPS09SH} to the multimodal scenarios. CMSSH \cite{cite:CVPR10CMSSH}, SCM \cite{cite:AAAI14SCM} and QCH \cite{cite:IJCAI15QCH} are supervised methods, which require that if two points are known to be similar, then their corresponding hash codes from different modalities should be similar. Since supervised methods can explore the semantic labels to enhance the cross-modal correlations and reduce the semantic gap \cite{cite:TPAMI00SemanticGap}, they can achieve superior accuracy than unsupervised methods for cross-modal similarity search with shorter hash codes.

Most of previous cross-modal hashing methods based on shallow architectures cannot exploit \emph{nonlinear} correlation across different modalities. And the latest cross-modal deep models \cite{cite:PAMI14CMNN,cite:JMLR14MMDL,cite:VLDB14MSAE,cite:MM14CorrAE} have shown that deep models can capture nonlinear cross-modal correlations more effectively. However, it remains unclear how to maximize cross-modal correlation and control quantization error in a hybrid deep architecture using well-specified loss functions. 

In this work, we further extend existing deep hashing methods \cite{cite:AAAI14CNNH,cite:CVPR15DNNH} to cross-modal retrieval by exploiting two key problems: (1) control the quantization error in a principled way, and (2) devise a more principled loss to link the pairwise Hamming distances with the pairwise similarity labels and to reduce the gap between Hamming distance and cosine distance. These crucial improvements constitute the proposed Correlation Hashing Network (CHN) approach.

\section{Correlation Hashing Network}\label{section:CHN}
%For ease of presentation, we describe the CHN approach with only two modalities, i.e. image and text, which can be readily extended to multiple modalities. 
In cross-modal retrieval, the database consists of objects from one modality and the query consists of objects from another modality. We uncover the correlation structure underlying different modalities by learning from a training set of $n$ bimodal objects $\{{\bm o}_i = ({\bm x}_i, {\bm y}_i)\}_{i=1}^n$, where ${\bm x}_i \in \mathbb{R}^{d_x}$ denotes the $d_x$-dimensional feature vector of the image modality, and ${\bm y}_i \in \mathbb{R}^{d_y}$ denotes the $d_y$-dimensional feature vectors of the text modality, respectively.
Some pairs of the bimodal objects are associated with similarity labels $s_{ij}$, where $s_{ij} = 1$ implies ${\bm o}_i$ and ${\bm o}_j$ are similar and $s_{ij} = -1$ indicates ${\bm o}_i$ and ${\bm o}_j$ are dissimilar. In supervised hashing, $\mathcal{S} = \{s_{ij}\}$ is usually constructed from the semantic labels of data points or the relevance feedback from click-through data. The goal of CHN is to jointly learn two modality-specific hashing functions $f_x\left( {\bm{x}} \right):{\mathbb{R}^{{d_x}}} \mapsto {\left\{ { - 1,1} \right\}^b}$ and $f_y\left( {\bm{y}} \right):{\mathbb{R}^{{d_y}}} \mapsto {\left\{ { - 1,1} \right\}^b}$ which respectively encode each unimodal point ${\bm x}$ and ${\bm y}$ in compact $b$-bit hash code ${\bm h}_x = f_x({\bm x})$ and ${\bm h}_y = f_y({\bm y})$ such that the similarity information conveyed in the given bimodal object pairs ${\cal S}$ is preserved.

The proposed Correlation Hashing Network (CHN) is a deep architecture for supervised learning to hash, shown in Figure~\ref{fig:CHN}. The architecture accepts input in a pairwise form $({\bm o}_i, {\bm o}_j, s_{ij})$ and processes them through the deep representation learning and hash coding pipeline: (1) an image network with multiple convolution-pooling layers to extract good image representations, and a text network with several fully-connected layers to extract good text representations; (2) a fully-connected hashing layer to generate modality-specific compact hash codes; (3) a squared cosine loss layer for capturing both cross-modal correlation and within-modal correlation; and (4) a new cosine quantization loss for controlling the quality of the binarized hash codes.

\begin{figure*}[tbp]
  \centering
  \includegraphics[width=0.95\textwidth]{CDH_Structure.pdf}
  \caption{Correlation Hashing Network (CHN), a hybrid deep architecture for cross-modal retrieval. The image network consists of multiple convolution-pooling layers for learning image representations, and the text network consists of multiple fully-connected layers for learning text representations. Two networks are coupled by  (1) a fully-connected hashing layer $fch$ for learning hash codes, (2) a squared cosine loss for capturing both cross-modal and within-modal correlations, and (3) a cosine quantization loss for controlling quality of hash codes.}
  \label{fig:CHN}
  \vspace{-10pt}
\end{figure*}

\subsection{Hybrid Deep Architecture}
Cross-modal data have significantly different statistical properties, which makes it very challenging to capture the correlation structures across modalities based on low-level features. Recently, it has been witnessed that deep learning methods \cite{cite:TPAMI13DL}, such as deep convolutional networks (CNNs) \cite{cite:NIPS12CNN} and deep recurrent networks (RNNs) \cite{cite:CVPR15RNN}, have made performance breakthroughs on many real-world recognition problems. Deep architectures are very powerful for maximizing the cross-modal correlation since they can extract multilayer feature representations with different abstraction levels \cite{cite:ICML13DCCA,cite:JMLR14MMDL,cite:MM14CorrAE}. We leverage the power of deep networks for cross-modal correlation maximization by proposing a hybrid deep architecture (CHN) as illustrated in Figure \ref{fig:CHN}, which constitutes an image network and a text network.

For the image network, we start with the AlexNet \cite{cite:NIPS12CNN}, a deep convolutional network (CNN) architecture comprised of five convolutional layers ($conv1$--$conv5$) and three fully connected layers ($fc6$--$fc8$). Each $fc$ layer $\ell$ learns a nonlinear mapping ${\bm{u}}_i^{{\ell}} = a_x^{{\ell}}\left( {{\bm{W}}_x^{\ell}{\bm{u}}_i^{{{\ell} - 1}} + {\bm{b}}_x^{\ell}} \right)$, where ${\bm{u}_i^{{\ell}}}$ is the $\ell$-th layer hidden representation of image $\bm{x}_i$, $\bm{W}_x^{\ell}$ and $\bm{b}_x^{\ell}$ are the weight and bias parameters of the $\ell$-th layer, and $a_x^{\ell}$ is the activation function, taken as rectifier units (ReLU) $a_x^{\ell}(\bm{x}) = \max(\bm{0}, \bm{x})$ for all hidden layers $conv1$--$fc7$. For hash learning, we replace the $fc8$ layer (the softmax classifier layer in the original AlexNet) with a new $fch$ layer of $b$ hidden units, which transforms the $fc7$ representation to $b$-dimensional hash coding by ${\bm h}^x_{i} = \operatorname{sgn} ({\bm u}_i^{l})$, where $l=8$ is the number of layers and ${\bm u}_i^{l}$ is the hidden representation of the $fch$ layer. To reduce the gap between the $fch$ layer representation ${\bm u}_i^{l}$ and the binary hash codes ${\bm h}_i^{x}$, we first squash the output to be within $[-1,1]$ by utilizing the hyperbolic tangent (tanh) activation function $a_x^{l}({\bm x}) = \tanh({\bm x})$. 

For the text network, we adopt Multilayer Perceptions (MLP) \cite{cite:MIT86MLP}, a widely-used architecture by the natural language processing (NLP) community. The MLP consists of  three fully-connected layers, which are respectively named as $fc6$, $fc7$ and $fc8$ to keep consistent with the image network. Each $fc$ layer $\ell$ learns a nonlinear mapping ${\bm{v}}_i^{{\ell }} = a_y^{{\ell }}\left( {{\bm{W}}_y^{\ell }{\bm{v}}_i^{{{\ell } - 1}} + {\bm{b}}_y^{\ell }} \right)$, where ${\bm{v}_i^{{\ell }}}$ is the $\ell$-th layer hidden representation of text $\bm{y}_i$, $\bm{W}_y^{\ell }$ and $\bm{b}_y^{\ell}$ are the weight and bias parameters of the $\ell$-th layer, and $a_y^{\ell}$ is the activation function, taken as rectifier units (ReLU) for the hidden layer $fc6$ and $fc7$. For hash learning, we replace $fc8$ with a $fch$ layer of $b$ hidden units, which transforms the $fc7$ representations to $b$-dimensional hash coding by ${\bm h}^y_{i} = \operatorname{sgn} ({{\bm v}_i^{l}})$, where ${\bm v}_i^{l}$ is the hidden representation of the $fch$ layer. To reduce the gap between the $fch$ layer representation ${\bm v}_i^{l}$ and the binary hash codes ${\bm h}_i^{y}$, we also use $\rm{tanh}$ as the activation function of the $fch$ layer for generating more compact hash codes. It is worth noting that, besides standard text representation methods such as tag occurrence and term-frequency inverse document frequency (TF-IDF), more advanced representation learning methods have been developed for natural language understanding, such as Word2Vec \cite{cite:NIPS13Word2Vec} and Paragraph Vector \cite{cite:Doc2Vec}. These advanced methods can be plugged into the MLP for learning more powerful text representations. In this paper, we keep our method simple and extendible by using tag occurrence as text features and MLP as the text network, while leaving advanced methods for future study.

\subsection{Squared Cosine Loss}
For a pair of binary codes ${\bm h}^x_i$ and ${\bm h}^y_j$, there is a nice relationship between their Hamming distance $\mathrm{dist}_H(\cdot,\cdot)$ and inner product $\langle \cdot,\cdot \rangle$: ${\textrm{dis}}{{\text{t}}_H}\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right) = \frac{1}{2}\left( {b - \left\langle {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right\rangle } \right)$. Therefore, we can employ the inner product as a reasonable surrogate of the Hamming distance to quantify the pairwise similarity. However, the approximation accuracy of such a surrogate for continuous representations ${\bm u}_i$ and ${\bm v}_j$ (note that ${\bm h}^x_i = \operatorname{sgn}({\bm u}_i)$ and ${\bm h}^y_i = \operatorname{sgn}({\bm v}_i)$) will be unreliable if their vector lengths are very different, i.e. $\frac{1}{2}\left( {b - \left\langle {{{\bm{u}}_i},{{\bm{v}}_j}} \right\rangle } \right)$ will no longer be an accurate surrogate of ${\textrm{dis}}{{\text{t}}_H}\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right)$. Figure~\ref{fig:CHN_Angular} shows such an undesirable case, where points $1$ and $2$ (in red) have very different vector lengths and hence very large Euclidean distance, but their Hamming distance is $0$ because they are assigned with the same binary code $(1,-1,1)$. Such a gap between Hamming distance and inner product has raised a misspecification issue of existing inner product based deep hashing methods \cite{cite:CVPR12KSH,cite:AAAI14CNNH,cite:CVPR15DNNH}.

To close the gap between Hamming distance and inner product for continuous embeddings, we note that for a pair of binary codes ${\bm h}^x_i$ and ${\bm h}^y_j$, there is another nice relationship between their Hamming distance $\mathrm{dist}_H(\cdot,\cdot)$ and the cosine distance $\cos (\cdot,\cdot)$: ${\textrm{dis}}{{\text{t}}_H}\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right) = \frac{b}{2}\left( {1 - \cos\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right) } \right)$, where $\cos \left( {{{\bm{u}}_i},{{\bm{v}}_j}} \right) = \frac{\left\langle {{\bm{u}}_i,{\bm{v}}_j} \right\rangle} {\left\| {{\bm{u}}_i} \right\|\left\| {{\bm{v}}_j} \right\|}$, and $\|\cdot\|$ is the vector length. Since cosine distance can mitigate the diversity of vector lengths and make the continuous representations ${\bm u}_i$ and ${\bm v}_j$ lie on the unit sphere (only the angle between them is measured), it makes $\frac{b}{2}\left( {1 - \cos \left( {{{\bm{u}}_i},{{\bm{v}}_j}} \right) } \right)$ a much more accurate surrogate of ${\textrm{dis}}{{\text{t}}_H}\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right)$. As can be seen in Figure~\ref{fig:CHN_Angular}, the cosine distance between points $1$ and $2$ (in red) is closer to zero and thus better approximates their Hamming distance. Hence in this paper, we opt to employ the cosine distance as a good surrogate of the Hamming distance, which leads to a novel squared cosine loss for correlation maximization.

\begin{figure}[tbp]
  \centering
  \includegraphics[width=0.95\columnwidth]{CDN_Angular.pdf}
  \vspace{-10pt}
  \caption{Justifications of the proposed squared cosine loss and cosine quantization loss. (1) Similar points $1$ and $2$ (in red): large Euclidean distance (bad Hamming surrogate) but small cosine distance (better Hamming surrogate). (2) Similar points $3$ and $4$ (in purple): small cosine distance but large Hamming distance (the gap between cosine and Hamming). (3) Similar points $5$ and $6$ (in yellow): small cosine distance and small Hamming distance (the gap is closed by cosine quantization loss). \emph{Best viewed in color}.}
  \label{fig:CHN_Angular}
  \vspace{-10pt}
\end{figure}

\subsubsection{Cross-Modal Correlation}
To maximize the cross-modal correlation, we propose the following criterion: for each pair of objects $({\bm o}_i, {\bm o}_j, s_{ij})$, if $s_{ij} = 1$, indicating that ${\bm o}_i$ and $\bm{o}_j$ are similar, then their binary hash codes must be similar across different modalities (image and text), i.e. the Hamming distance should satisfy $d_H({\bm h}^x_i$, ${\bm h}^y_j) \rightarrow 0$ and $d_H({\bm h}^y_i$, ${\bm h}^x_j) \rightarrow 0$, which implies the cosine distance should satisfy $\cos({\bm u}_i$, ${\bm v}_j) \rightarrow 1$ and $\cos({\bm v}_i$, ${\bm u}_j) \rightarrow 1$. Correspondingly, if $s_{ij} = -1$, indicating that ${\bm o}_i$ and $\bm{o}_j$ are dissimilar, then by derivation, the cosine distance should satisfy $\cos({\bm u}_i$, ${\bm v}_j) \rightarrow -1$ and $\cos({\bm v}_i$, ${\bm u}_j) \rightarrow -1$. It is very important to note that, for other distance metrics (e.g. inner product, Euclidean distance, etc), it is very difficult to devise such a well-specified learning criterion because these distances are not good surrogates of the Hamming distance. The above learning criterion leads to a novel squared cosine loss for maximizing the cross-modal correlation as follows,
\begin{equation}\label{eqn:CrossL}
	{C_{xy}} = \sum\limits_{{s_{ij}} \in \mathcal{S}} {{{{\left( {{s_{ij}} - \cos \left( {{{\bm{u}}_i},{{\bm{v}}_j}} \right)} \right)}^2} + {{\left( {{s_{ij}} - \cos \left( {{{\bm{v}}_i},{{\bm{u}}_j}} \right)} \right)}^2}}},
\end{equation}
The range of cosine distance ${\cos \left( {{{\bm{u}}_i},{{\bm{v}}_j}} \right)} \in [-1,1]$ is consistent with binary similarity labels $s_{ij}\in\{-1,1\}$, making the squared cosine loss in Equation~\eqref{eqn:CrossL} a well-specified loss for preserving the pairwise similarity information conveyed in $\mathcal{S}$. The squared cosine loss is particularly powerful for cross-modal correlation analysis, since the vector lengths are very diverse in different modalities and may make other distance metrics (e.g. Euclidean distance) misspecified. In real retrieval systems, cosine distance is widely adopted to mitigate the diversity of vector lengths and significantly improve the retrieval quality, but it has not been used in deep learning to hash methods for the cross-modal retrieval \cite{cite:arxiv14hashsurvey}. 

\subsubsection{Within-Modal Correlation}
Only maximizing the cross-modal correlation is not sufficient for learning high-quality hash codes for cross-modal retrieval. The justifications are two folds: (1) Theoretically, each modality may have its modality-specific structure (image or text), which cannot be well captured by exploring its correlation with another modality; (2) Technically, deep neural networks cannot receive sufficient training if within-modal knowledge structure is not explored (due to gradient vanishing issue). Hence, we further maximize the within-modal correlation using the squared cosine loss as follows,
\begin{equation}\label{eqn:IntraL}
	\begin{gathered}
    {C_{xx}} = \sum\limits_{{s_{ij}} \in \mathcal{S}} {{{\left( {{s_{ij}} - \cos \left( {{{\bm{u}}_i},{{\bm{u}}_j}} \right)} \right)}^2}}, \\
    {C_{yy}} = \sum\limits_{{s_{ij}} \in \mathcal{S}} {{{\left( {{s_{ij}} - \cos \left( {{{\bm{v}}_i},{{\bm{v}}_j}} \right)} \right)}^2}}. \\
  \end{gathered}
\end{equation}
By maximizing the within-modal correlation, we can learn high-quality deep representations for both image and text by using the image network and text network, respectively. The high-quality modality-specific deep representations guarantee that the cross-modal correlation can be captured and enhanced by the hybrid deep architecture while the different convergence rates between the image network (CNNs) and the text network (MLPs) will not cause a training difficulty.

\subsection{Cosine Quantization Loss}
Although we justify that the cosine distance is a good surrogate of the Hamming distance by ${\textrm{dis}}{{\text{t}}_H}\left( {{{\bm{h}}^x_i},{{\bm{h}}^y_j}} \right) \approx \frac{b}{2}\left( {1 - \cos\left( {{{\bm{u}}_i},{{\bm{v}}_j}} \right) } \right)$, such an approximation may fail when two similar points ${\bm u}_i$ and ${\bm v}_j$ with $s_{ij} = 1$ (i.e. their cosine distance is small due to minimizing the squared cosine loss) lie on different sides of the hyperplane (i.e. their Hamming distance is large due to different signs of hash codes across the hyperplane). Figure~\ref{fig:CHN_Angular} shows such an undesirable case, where points $3$ and $4$ (in purple) have small cosine distance but large Hamming distance because they are assigned with different binary codes $(1,-1,1)$ and $(1,1,1)$, respectively. Such a gap between Hamming distance and cosine distance may result in an inaccurate surrogate approximation.

To close the gap between Hamming distance and cosine distance for continuous embeddings, we note that for a pair of continuous representations ${\bm u}_i$ and ${\bm v}_j$, if they are close (in cosine distance) to their signed codes ${\bm h}^x_i = \operatorname{sgn}({\bm u}_i)$ and ${\bm h}^y_i = \operatorname{sgn}({\bm v}_i)$ (i.e. far from the hyperplane), then they will lie in the same hypercube with high probability (i.e. have the same binary code and hence their Hamming distance is zero). Figure~\ref{fig:CHN_Angular} shows such a desirable case by points $5$ and $6$ (in yellow). Hence we need to maximize $\cos ({\bm u}_i, \operatorname{sgn}({\bm u}_i))$ and $\cos ({\bm v}_j, \operatorname{sgn}({\bm v}_j))$, which is equivalent to maximizing $\cos (|{\bm u}_i|, {\bm 1})$ and $\cos (|{\bm v}_j|, {\bm 1})$, respectively. The observation leads to a novel cosine quantization loss defined as follows,
\begin{equation}\label{eqn:QuanL}
\begin{gathered}
{Q_x} =  - \sum\limits_{{s_{ij}} \in \mathcal{S}} {\left( {\cos \left( {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right) + \cos \left( {\left| {{{\bm{u}}_j}} \right|,{\bm{1}}} \right)} \right)}, \\
{Q_y} =  - \sum\limits_{{s_{ij}} \in \mathcal{S}} {\left( {\cos \left( {\left| {{{\bm{v}}_i}} \right|,{\bm{1}}} \right) + \cos \left( {\left| {{{\bm{v}}_j}} \right|,{\bm{1}}} \right)} \right)}, \\
\end{gathered}
\end{equation}
where $\bm{1} \in \mathbb{R}^b$ is the vector of ones. Note that, minimizing the cosine quantization loss will not only close the gap between the Hamming distance and cosine distance, but also lead to lower quantization error when binarizing the continuous representations ${\bm u}_i \in \mathbb{R}^b$ and ${\bm v}_j \in \mathbb{R}^b$ to hash codes ${\bm h}^x_i = \operatorname{sgn}({\bm u}_i)\in \{-1,1\}^b$ and ${\bm h}^y_j = \operatorname{sgn}({\bm v}_j) \in \{1,-1\}^b$. This advantage enables us to learn high-quality hash codes.

\subsection{Joint Optimization Problem}
We perform simultaneous representation learning and hash coding by jointly maximizing the cross-modal/within-modal correlations and controlling the quantization error. Integrating \eqref{eqn:CrossL}--\eqref{eqn:QuanL} into a joint optimization problem yields
\begin{equation}\label{eqn:Unified}
\mathop {\min }\limits_\Theta  L = {C_{xy}} + \lambda \left( {{C_{xx}} + {C_{yy}}} \right) + \gamma \left( {{Q_x} + {Q_y}} \right),
\end{equation}
where $\Theta \triangleq \left\{ {{{\bm{W}}^\ell },{{\bm{b}}^\ell }} \right\}$ is the set of network parameters, $\lambda$ is the penalty parameter to control the relative importance of within-modal squared cosine loss, and $\gamma$ is the penalty parameter for trading off the relative importance of the cosine quantization loss. Through joint optimization problem \eqref{eqn:Unified}, we can achieve optimal learning of compact hash codes for cross-modal retrieval, e.g. text query on image database. Finally, we can obtain $b$-bit binary codes by simple quantization ${\bm h}_x \leftarrow \mathrm{sgn}({\bm u})$ and ${\bm h}_y \leftarrow \mathrm{sgn}({\bm v})$, where $\mathrm{sgn}({\bm z})$ is the sign function on vectors that for $i = 1,\ldots,b$, $\mathrm{sgn}(z_i) = 1$ if $z_i > 0$, otherwise $\mathrm{sgn}(z_i) = -1$. It is worth noting that, since we have minimized the cosine quantization error in \eqref{eqn:Unified} during training, this final binarization step will incur very small loss of retrieval quality as validated empirically.

\section{Algorithm and Analysis}
\subsection{Learning Algorithm}
We derive the learning algorithms for the CHN model in Equation~\eqref{eqn:Unified}, and show rigorously that both squared cosine loss and cosine quantization loss can be optimized efficiently through the standard back-propagation (BP) procedure. For notation brevity, we define the pointwise cost as
\vspace{-10pt}
\begin{equation}\label{eqn:pairwise}
    \footnotesize
    \begin{aligned}
    	{L_i} &\triangleq \sum\limits_{j:{s_{ij}} \in \mathcal{S}} { {C_{ij}^{xy} + \lambda (C_{ij}^{xx} + C_{ij}^{yy}) + \gamma (Q_{ij}^x + Q_{ij}^y)} }\\
     & = \sum\limits_{j:{s_{ij}} \in \mathcal{S}}  {{{{\left( {{s_{ij}} - \cos \left( {{{\bm{u}}_i^{l}},{{\bm{v}}_j^{l}}} \right)} \right)}^2} + {{\left( {{s_{ij}} - \cos \left( {{{\bm{v}}_i^{l}},{{\bm{u}}_j^{l}}} \right)} \right)}^2}}}\\
     & + \lambda \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {{{{{\left( {{s_{ij}} - \cos \left( {{{\bm{u}}_i^{l}},{{\bm{u}}_j^{l}}} \right)} \right)}^2} + {{\left( {{s_{ij}} - \cos \left( {{{\bm{v}}_i^{l}},{{\bm{v}}_j^{l}}} \right)} \right)}^2}}} }\\
     & - \gamma \sum\limits_{j:{s_{ij}} \in \mathcal{S}} \left( {\left( {\cos \left( {\left| {{{\bm{u}}_i^{l}}} \right|,{\bm{1}}} \right) + \cos \left( {\left| {{{\bm{u}}_j^{l}}} \right|,{\bm{1}}} \right)} \right)}  \right)\\
     & - \gamma \sum\limits_{j:{s_{ij}} \in \mathcal{S}} \left( {\left( {\cos \left( {\left| {{{\bm{v}}_i^{l}}} \right|,{\bm{1}}} \right) + \cos \left( {\left| {{{\bm{v}}_j^{l}}} \right|,{\bm{1}}} \right)} \right)}  \right),\\
    \end{aligned}
    \normalsize
\end{equation}
where ${\bm u}_i = {\bm u}^l_i$ and ${\bm v}_j = {\bm v}^l_i$ denote the continuous representations of the hybrid network. We derive the gradient of point-wise cost $L_i$ w.r.t. ${\bm W}_{x,k}^{\ell}$, the network parameter of the $k$-th unit in the $\ell$-th layer for the image network (the derivation of the text network is the same and is omitted) as
\vspace{-10pt}
\begin{equation}
    \footnotesize
    \begin{aligned}
    \frac{{\partial {L_i}}}{{\partial {\bm{W}}_{x,k}^{\ell} }} &= \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left( {\frac{{\partial {C^{xy}_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }} + \lambda \frac{{\partial {C^{xx}_{ij} + C^{yy}_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }} }\right)}\\
    & + \sum\limits_{j:{s_{ij}} \in \mathcal{S}}  {\left( { \gamma \frac{{\partial {Q^x_{ij}+Q^y_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }}} \right)} \\
    &= \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left( {\frac{{\partial {C^{xy}_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }} + \lambda \frac{{\partial {C^{xx}_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }} + \gamma \frac{{\partial {Q^x_{ij}}}}{{\partial {\bm{W}}_{x,k}^{\ell} }}} \right)} \\
    &= \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left( {\frac{{\partial {C^{xy}_{ij}}}}{{\partial \hat u_{ik}^{\ell}}} + \lambda \frac{{\partial {C^{xx}_{ij}}}}{{\partial \hat u_{ik}^{\ell}}} + \gamma \frac{{\partial {Q^x_{ij}}}}{{\partial \hat u_{ik}^{\ell}}}} \right) \frac{{\partial \hat u_{ik}^{\ell} }}{{\partial {\bm{W}}_{x,k}^{\ell} }}} \\
     &= \delta _{x,ik}^{\ell} {\bm{u}}_{i}^{\ell  - 1} , \\ 
    \end{aligned}
    \normalsize
\end{equation}
where ${\bm{\hat u}}_{i}^{\ell}  = {{{\bm{W}}_x^{\ell}}{\bm{u}}_{i}^{\ell  - 1} + {{\bm{b}}_x^{\ell}}}$ is $\ell$-th layer output before activation $a_x^{\ell}(\cdot)$, $\delta _{x,ik}^{\ell}  \triangleq \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left( {\frac{{\partial {C^{xy}_{ij}}}}{{\partial \hat u_{ik}^{\ell}}} + \lambda \frac{{\partial {C^{xx}_{ij}}}}{{\partial \hat u_{ik}^{\ell}}} + \gamma \frac{{\partial {Q^x_{ij}}}}{{\partial \hat u_{ik}^{\ell}}}}  \right)} $ is the point-wise \emph{residual} term that measures how much the $k$-th unit in the $\ell$-th layer is responsible for the error of point ${\bm x}_i$ in the network output. For an output unit $k$, we can measure the difference between the network's activation and the true target value, and use that to define the residual $\delta _{x,ik}^{l}$ as
%\begin{equation}\label{eqn:deltaL}
%    \footnotesize
%  \begin{aligned}
%        \delta _{x,ik}^{l} &= \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left[ {\left( {\frac{{\left\langle {{\bm{u}}_i^{l},{\bm{v}}_j^{l}} \right\rangle }}{{\left\| {{\bm{u}}_i^{l}} \right\|\left\| {{\bm{v}}_j^{l}} \right\|}} - {s_{ij}}} \right)v_{jk}^{l}} \right]{{\dot a}^{l}}\left( {\hat u_{ik}^{l}} \right)}  \\
%    &+ \lambda \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left[ {\left( {\frac{{\left\langle {{\bm{u}}_i^{l},{\bm{u}}_j^{l}} \right\rangle }}{{\left\| {{\bm{u}}_i^{l}} \right\|\left\| {{\bm{u}}_j^{l}} \right\|}} - {s_{ij}}} \right)u_{jk}^{l}} \right]{{\dot a}^{l}}\left( {\hat u_{ik}^{l}} \right)}  \\
%    &+ \gamma \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\frac{1}{\sqrt{b} \left\| {\bm{u}}_j^{l} \right\|}  \operatorname{sgn} \left( {u_{ik}^{l}} \right){{\dot a}_x^{l}}\left( {\hat u_{ik}^{l}} \right)}, \\
%    \end{aligned}
%    \normalsize
%\end{equation}
\vspace{-10pt}
\begin{equation}\label{eqn:deltaL}
	\footnotesize
  \begin{aligned}
  & \delta _{x,ik}^l = (\sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left[ {\left( {\frac{{\left\langle {{\bm{u}}_i^l,{\bm{v}}_j^l} \right\rangle }}{{\left\| {{\bm{u}}_i^l} \right\|\left\| {{\bm{v}}_j^l} \right\|}} - {s_{ij}}} \right)\left( {\frac{{v_{jk}^l}}{{\left\| {{\bm{u}}_i^l} \right\|\left\| {{\bm{v}}_j^l} \right\|}} - \frac{{u_{ik}^l\left\langle {{\bm{u}}_i^l,{\bm{v}}_j^l} \right\rangle }}{{{{\left\| {{\bm{u}}_i^l} \right\|}^3}\left\| {{\bm{v}}_j^l} \right\|}}} \right)} \right]} \\
   &+ \lambda \sum\limits_{j \ne i:{s_{ij}} \in \mathcal{S}} {\left[ {\left( {\frac{{\left\langle {{\bm{u}}_i^l,{\bm{u}}_j^l} \right\rangle }}{{\left\| {{\bm{u}}_i^l} \right\|\left\| {{\bm{u}}_j^l} \right\|}} - {s_{ij}}} \right)\left( {\frac{{u_{jk}^l}}{{\left\| {{\bm{u}}_i^l} \right\|\left\| {{\bm{u}}_j^l} \right\|}} - \frac{{u_{ik}^l\left\langle {{\bm{u}}_i^l,{\bm{u}}_j^l} \right\rangle }}{{{{\left\| {{\bm{u}}_i^l} \right\|}^3}\left\| {{\bm{u}}_j^l} \right\|}}} \right)} \right]}  \\
   & + \gamma \sum\limits_{j:{s_{ij}} \in \mathcal{S}} {\left[ {\frac{{\operatorname{sgn} \left( {{u_{ik}}} \right)}}{{\sqrt b \left\| {{{\bm{u}}_i}} \right\|}} - \frac{{{u_{ik}}\left\langle {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right\rangle }}{{\sqrt b {{\left\| {{{\bm{u}}_i}} \right\|}^3}}}} \right]} ) \cdot 2 {{\dot a}^l}\left( {\hat u_{ik}^l} \right) \\ 
	\end{aligned}
	\normalsize
\end{equation}
where $l=8$ denotes the index of the output layer, and ${\dot a}_x^{l}(\cdot)$ is the derivative of the $l$-th layer activation function. For a hidden unit $k$ in the $(\ell-1)$-th layer, we compute the residual $\delta _{x,ik}^{\ell-1}$ based on a weighted average of the errors of all the units ${k'}=1,\ldots,n_{\ell-1}$ in the $(\ell-1)$-th layer that use ${\bm u}_i^{\ell-1}$ as an input, which is just consistent with the BP procedure,
\begin{equation}\label{eqn:deltaEll}
    \delta _{x,ik}^{\ell  - 1} = \left( {\sum\limits_{k' = 1}^{{n_{\ell  - 1}}} {\delta _{x,ik'}^{\ell} W_{x,kk'}^{\ell  - 1}} } \right){{\dot a}_x^{\ell  - 1}}\left( {\hat u_{ik}^{\ell  - 1}} \right) ,
\end{equation}
where $n_{\ell-1}$ is number of units in $(\ell-1)$-th layer. The residuals in all layers can be computed by back-propagation. 

An important property of the proposed algorithm is that, only computing the residual of the output layer involves the pairwise summation as in Equation~\eqref{eqn:deltaL}. For all hidden layers, all the residuals can be simply computed recursively by Equation~\eqref{eqn:deltaEll}, which does not involve pairwise summation. Hence we do not need to modify the implementation of BP in all hidden layers $1\le\ell\le l-1$. We only modify standard BP by replacing the output residual with Equation~\eqref{eqn:deltaL}.

Since the only difference between standard BP and our algorithm is Equation~\eqref{eqn:deltaL}, we analyze the computational complexity based on Equation~\eqref{eqn:deltaL}. Denote the number of similarity pairs $\mathcal{S}$ available for training as $|\mathcal{S}|$, then it is easy to verify that the computational complexity is linear $O(|\mathcal{S}|)$.

\subsection{Theoretical Analysis}
We elaborate the connection between our work and Iterative Quantization (ITQ) \cite{cite:CVPR11ITQ}, the seminal work that considers the quantization error in an iterative Procrustean procedure. In ITQ, point-wise quantization error is defined as follows,
\begin{equation}\label{eqn:ITQ}
    {Q_{{\text{ITQ}}}} = {{{\left\| {{{\bm{u}}_i} - \operatorname{sgn} \left( {{{\bm{u}}_i}} \right)} \right\|}^2}} .
\end{equation}
We only consider point-wise quantization error for brevity, while taking the summation for all training points (pairs) yields the overall quantization error similar to Equation~\eqref{eqn:QuanL}.

\begin{theorem}[Upper Bound]\label{theorem:bound}
    The proposed cosine quantization loss in \eqref{eqn:QuanL} is proportional to an upper bound of the ITQ quantization error in \eqref{eqn:ITQ} (up to a constant $2b$),
    \begin{equation}\label{eqn:bound}
        {\left\| {{{\bm{u}}_i} - \operatorname{sgn} \left( {{{\bm{u}}_i}} \right)} \right\|^2} \le 2b - 2b\cos \left( {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right).
    \end{equation}
\end{theorem}

\begin{proof}
    It follows from triangle inequality ${\left\| {\left| {{{\bm{u}}_i}} \right| - {\bm{1}}} \right\|^2} = {\left\| {\left| {{{\bm{u}}_i}} \right| - \left| {\operatorname{sgn} \left( {{{\bm{u}}_i}} \right)} \right|} \right\|^2} \leqslant {\left\| {{{\bm{u}}_i} - \operatorname{sgn} \left( {{{\bm{u}}_i}} \right)} \right\|^2}$. Since ${\bm u}_i$ and $\operatorname{sgn}({\bm u}_i)$ have the same sign, the last equality always holds, hence $ {\left\| {{{\bm{u}}_i} - \operatorname{sgn} \left( {{{\bm{u}}_i}} \right)} \right\|^2} = {\left\| {\left| {{{\bm{u}}_i}} \right| - {\bm{1}}} \right\|^2}$. Because we use hyperbolic tangent ($\tanh$) as the activation function for the $fch$ layer, $|{\bm u}_i| \le {\bm 1}$, $\|{\bm u}_i\|^2 \le b$, and $\left\langle {\left| {{{\bm{h}}_i}} \right|,{\bm{1}}} \right\rangle  \geqslant 0$. Hence,
    \begin{equation}
    	\begin{aligned}
        {\left\| {\left| {{{\bm{u}}_i}} \right| - {\bm{1}}} \right\|^2} & = {\left\| {{{\bm{u}}_i}} \right\|^2}{\left\| {\frac{{\left| {{{\bm{u}}_i}} \right|}}{{\left\| {{{\bm{u}}_i}} \right\|}} - \frac{{\bm{1}}}{{\left\| {{{\bm{u}}_i}} \right\|}}} \right\|^2} \\
				& \leqslant b{\left\| {\frac{{\left| {{{\bm{u}}_i}} \right|}}{{\left\| {{{\bm{u}}_i}} \right\|}} - \frac{{\bm{1}}}{{\left\| {{{\bm{u}}_i}} \right\|}}} \right\|^2} = 2b\left( {1 - \frac{{\left\langle {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right\rangle }}{{{{\left\| {{{\bm{u}}_i}} \right\|}^2}}}} \right) \\
				& \leqslant 2b\left( {1 - \frac{{\left\langle {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right\rangle }}{{\left\| {{{\bm{u}}_i}} \right\|\sqrt b }}} \right) = 2b - 2b\cos \left( {\left| {{{\bm{u}}_i}} \right|,{\bm{1}}} \right).  \\ 								
			\end{aligned}
    \end{equation}
\end{proof}
Theorem~\ref{theorem:bound} reveals that the cosine quantization loss \eqref{eqn:QuanL} is a reasonable criterion for hash learning. Different from all existing methods, the proposed cosine quantization loss is jointly optimized with the squared cosine loss in a hybrid deep network, which can yield better hashing schemes by jointly (1) reducing the gap between Hamming distance and cosine distance and (2) controlling the binary coding error.

\section{Experiments}\label{section:Experiments}
We conduct extensive experiments to evaluate the efficacy of the proposed CHN model with several state of the art hashing methods on widely-used benchmark datasets. The codes and configurations will be available online.

\begin{table*}[tb]
    \addtolength{\tabcolsep}{4.0pt} 
    \centering 
    \caption{Mean Average Precision (MAP) Comparison of Two Cross-modal Retrieval Tasks on Two Datasets}
    \label{table:MAP}
    \small
    \begin{tabular}{c|c|cccc|cccc}
        \Xhline{1.0pt}
        \multirow{2}{20pt}{\centering Task} & \multirow{2}{20pt}{\centering Method} & \multicolumn{4}{c|}{NUS-WIDE} & \multicolumn{4}{c}{MIR-Flickr}\\
        \cline{3-10}
        & & 8 bits & 16 bits  & 32 bits  & 64 bits & 8 bits & 16 bits  & 32 bits  & 64 bits \\
        \hline
        \multirow{8}{30pt}{\centering $ I \rightarrow T$} 
        & CMSSH & 0.3950 & 0.4052 & 0.4076 & 0.3516 & 0.5076 & 0.5272 & 0.5357 & 0.5219 \\
        & CVH & 0.4588 & 0.4713 & 0.4743 & 0.4740 & 0.6091 & 0.6225 & 0.6364 & 0.6199 \\
        & IMH & 0.4345 & 0.4399 & 0.4203 & 0.4115 & 0.5449 & 0.5646 & 0.5936 & 0.5539 \\
        & SCM & 0.4693 & 0.4648 & 0.4619 & 0.4851 & 0.6361 & 0.6493 & 0.6495 & 0.6440\\
        & QCH & 0.4765 & 0.4895 & 0.5050 & 0.5125 & 0.6452 & 0.6523 & 0.6685 & 0.6721 \\
        & CorrAE & 0.4398 & 0.4522 & 0.4699 & 0.4944 & 0.6301 &  0.6329 & 0.6357 & 0.6401 \\
        & CM-NN & \underline{0.6035} & \underline{0.6255} & \underline{0.6424} & \underline{0.6514} & \underline{0.7154} & \underline{0.7288} & \underline{0.7415} & \underline{0.7541} \\
        \cline{2-10}
        & CHN & \textbf{0.7126} & \textbf{0.7692} & \textbf{0.8220} & \textbf{0.8277} & \textbf{0.7890} & \textbf{0.8430} & \textbf{0.8571} & \textbf{0.8572} \\
        \hline
        \multirow{8}{30pt}{\centering $ T \rightarrow I$} 
        & CMSSH & 0.3783 & 0.3499 & 0.3944 & 0.4015 & 0.5868 & 0.5732 & 0.6176 & 0.6323 \\
        & CVH & 0.5598 & 0.5217 & 0.5129 & 0.4875 & 0.5972 & 0.6032 & 0.5738 & 0.5794 \\
        & IMH & 0.4380 & 0.4582 & 0.4186 & 0.4051 & 0.5374 & 0.5536 & 0.5513 & 0.5583 \\
        & SCM & 0.4449 & 0.4859 & 0.5105 & 0.5259 & 0.6037 & 0.5998 & 0.5805 & 0.6078\\
        & QCH & 0.5020 & 0.5195 & 0.5489 & 0.5622 & 0.6258 & 0.6425 & 0.6485 & 0.6528 \\
        & CorrAE & 0.4303 & 0.4501 & 0.4634 & 0.4880 & 0.6142 & 0.6198 & 0.6247 & 0.6431\\
        & CM-NN & \underline{0.6012} & \underline{0.6083} & \underline{0.6226} & \underline{0.6435} & \underline{0.6969} & \underline{0.7142} & \underline{0.7285} & \underline{0.7352}  \\
        \cline{2-10}
        & CHN & \textbf{0.7170} & \textbf{0.7617} & \textbf{0.7704} & \textbf{0.7649} & \textbf{0.7595} & \textbf{0.7631} & \textbf{0.7814} & \textbf{0.7886} \\
        \Xhline{1.0pt}
    \end{tabular}
    \normalsize
    \vspace{-4pt}
\end{table*}


\begin{figure*}[tbp]
    \centering
%    \subfigure[$ I \rightarrow T$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{pr_nuswide_1.eps}
%        \label{fig:pr_nuswide_1}
%    }
    \subfigure[$ I \rightarrow T$ on NUS-WIDE]{
        \includegraphics[width=0.23\textwidth]{pr_nuswide_2.eps}
        \label{fig:pr_nuswide_2}
    }
%    \subfigure[$ T \rightarrow I$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{pr_nuswide_3.eps}
%        \label{fig:pr_nuswide_3}
%    }
    \subfigure[$ T \rightarrow I$ on NUS-WIDE]{
        \includegraphics[width=0.23\textwidth]{pr_nuswide_4.eps}
        \label{fig:pr_nuswide_4}
    }
%    \\
%    \subfigure[$ I \rightarrow T$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{pr_flickr1m_1.eps}
%        \label{fig:pr_flickr1m_1}
%    }
    \subfigure[$ I \rightarrow T$ on Flickr]{
        \includegraphics[width=0.23\textwidth]{pr_flickr1m_2.eps}
        \label{fig:pr_flickr1m_2}
    }
%    \subfigure[$ T \rightarrow I$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{pr_flickr1m_3.eps}
%        \label{fig:pr_flickr1m_3}
%    }
    \subfigure[$ T \rightarrow I$ on Flickr]{
        \includegraphics[width=0.23\textwidth]{pr_flickr1m_4.eps}
        \label{fig:pr_flickr1m_4}
    }
    \caption{Precision-recall curves of the two cross-modal retrieval tasks on the NUS-WIDE and Flickr datasets with 32 bits hash codes.}
    \label{fig:pr}
    \vspace{-10pt}
\end{figure*}

\begin{figure*}[tbp]
    \centering
%    \subfigure[$ I \rightarrow T$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{prec_nuswide_1.eps}
%        \label{fig:prec_nuswide_1}
%    }
    \subfigure[$ I \rightarrow T$ on NUS-WIDE]{
        \includegraphics[width=0.23\textwidth]{prec_nuswide_2.eps}
        \label{fig:prec_nuswide_2}
    }
%    \subfigure[$ T \rightarrow I$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{prec_nuswide_3.eps}
%        \label{fig:prec_nuswide_3}
%    }
    \subfigure[$ T \rightarrow I$ on NUS-WIDE]{
        \includegraphics[width=0.23\textwidth]{prec_nuswide_4.eps}
        \label{fig:prec_nuswide_4}
    }
%    \\
%    \subfigure[$ I \rightarrow T$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{prec_flickr1m_1.eps}
%        \label{fig:prec_flickr1m_1}
%    }
    \subfigure[$ I \rightarrow T$ on Flickr]{
        \includegraphics[width=0.23\textwidth]{prec_flickr1m_2.eps}
        \label{fig:prec_flickr1m_2}
    }
%    \subfigure[$ T \rightarrow I$ @ 16 bits]{
%        \includegraphics[width=0.23\textwidth]{prec_flickr1m_3.eps}
%        \label{fig:prec_flickr1m_3}
%    }
    \subfigure[$ T \rightarrow I$ on Flickr]{
        \includegraphics[width=0.23\textwidth]{prec_flickr1m_4.eps}
        \label{fig:prec_flickr1m_4}
    }
    \caption{Precision@top R curves of the two cross-modal retrieval tasks on the NUS-WIDE and Flickr datasets with 32 bits hash codes.}
    \label{fig:prec}
\end{figure*}

\begin{table*}[tb]
    \addtolength{\tabcolsep}{4.0pt} 
    \centering 
    \caption{Mean Average Precision (MAP) Comparison of Two Cross-modal Retrieval Tasks of CHN variants on Two Datasets.}
    \label{table:EmpirMAP}
    \small
    \begin{tabular}{c|c|cccc|cccc}
        \Xhline{1.0pt}
        \multirow{2}{20pt}{\centering Task} & \multirow{2}{20pt}{\centering Method} & \multicolumn{4}{c|}{NUS-WIDE} & \multicolumn{4}{c}{MIR-Flickr}\\
        \cline{3-10}
        & & 8 bits & 16 bits  & 32 bits  & 64 bits & 8 bits & 16 bits  & 32 bits  & 64 bits \\
        \hline
        \multirow{4}{30pt}{\centering $ I \rightarrow T$} 
        & CHN-W & 0.3257 & 0.3718 & 0.3696 & 0.3580 & 0.5593 & 0.5676 & 0.5810 & 0.6069 \\
        & CHN-C &0.6214  & 0.6748 & 0.7235 & 0.7582 & 0.7584 & 0.7878 & 0.7942 & 0.7992\\
        & CHN-Q & \underline{0.6852}  & \underline{0.7445} & \underline{0.7890} & \underline{0.8035} & \underline{0.7671} & \underline{0.8056} & \underline{0.8173} & \underline{0.8229} \\
        & CHN & \textbf{0.7126} & \textbf{0.7692} & \textbf{0.8220} & \textbf{0.8277} & \textbf{0.7890} & \textbf{0.8430} & \textbf{0.8571} & \textbf{0.8572} \\
        \hline
        \multirow{4}{30pt}{\centering $ T \rightarrow I$} 
        & CHN-W & 0.3368 & 0.3500 & 0.3262 & 0.3868 & 0.4346 & 0.4649 & 0.4906 & 0.5203 \\
        & CHN-C & 0.6045 & 0.6327 & 0.6514 & 0.6790 & 0.7215 & 0.7349 & 0.7396 & 0.7581 \\
        & CHN-Q & \underline{0.6922} & \underline{0.7418} & \underline{0.7562} & \underline{0.7581} & \underline{0.7384} & \underline{0.7519} & \underline{0.7667} & \underline{0.7784} \\
        & CHN & \textbf{0.7170} & \textbf{0.7617} & \textbf{0.7704} & \textbf{0.7649} & \textbf{0.7595} & \textbf{0.7631} & \textbf{0.7814} & \textbf{0.7886} \\
        \Xhline{1.0pt}
    \end{tabular}
    \normalsize
\end{table*}

\subsection{Evaluation Setup}
We conduct our extensive evaluation on two benchmark cross-modal datasets: \textbf{NUS-WIDE} \cite{cite:CIVR09NusWide} and \textbf{MIR-Flickr} \cite{cite:ICMR08Flickr1M}.

\textbf{NUS-WIDE}\footnote{\scriptsize\url{http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm}} is a public web image dataset containing 269, 648 images downloaded from Flickr, together with the associated raw tags of these images.  There are 81 ground truth concepts manually annotated for search evaluation. Following prior works \cite{cite:MM13LCMH,cite:VLDB14MSAE}, we prune the original NUS-WIDE dataset to form a new dataset consisting of 195,834 image-text pairs that belong to one of the 21 most frequent concepts. All images are resized into 256$\times$256.

\textbf{MIR-Flickr} includes 1 million images associated with tags from Flickr, where 25,000 are labeled with 38 concepts. We resize images of this labeled subset into 256$\times$256. 

For our deep learning based approach CHN, we directly use the raw image pixels as the input. For the shallow learning based methods, we follow \cite{cite:CVPR12KSH,cite:CVPR15DNNH,cite:JMLR14MMDL} to represent each image in NUS-WIDE by a 500-dimensional bag-of-visual-words vector, and to represent each image in Flickr by a 3,857-dimensional vector concatenated by local SIFT feature, global GIST feature, and etc.
For text modality, all the methods use tag occurrence vectors as the input.
% 1000-dimensional feature vector for NUS-WIDE and 2000-dimensional feature vector for Flickr.
All image and text features are available at the datasets' website.
In NUS-WIDE, we randomly select 100 pairs per class as the test query set, 500 pairs per class as the training set and 50 pairs per class as the validation set. In Flickr, we randomly select 1000 pairs as the test query set, 4000 pairs as the training set and 1000 pairs as the validation set. The similarity pairs for training are randomly constructed using semantic labels: each pair is considered similar (dissimilar) if they share at least one (none) semantic label.

We evaluate and compare the retrieval accuracies of the proposed CHN with 7 state-of-the-art cross-modal hashing methods, including three unsupervised methods 
\textbf{IMH}\footnote{\scriptsize\url{http://staff.itee.uq.edu.au/shenht/UQ_IMH}} \cite{cite:SIGMOD13IMH}, 
\textbf{CVH}\footnote{\scriptsize\url{http://www.cse.ust.hk/~dyyeung/code/mlbe.zip}\label{foot:CMSSH}} \cite{cite:IJCAI11CVH} and 
\textbf{CorrAE}\footnote{\scriptsize{Since the original code is not publicly available, we implement it by ourselves.}\label{foot:SCM}} \cite{cite:MM14CorrAE}, 
and four supervised methods 
\textbf{CMSSH}\footref{foot:CMSSH} \cite{cite:CVPR10CMSSH}, 
\textbf{SCM}\footref{foot:SCM} \cite{cite:AAAI14SCM}, 
\textbf{QCH}\footref{foot:SCM} \cite{cite:IJCAI15QCH} and 
\textbf{CM-NN}\footref{foot:SCM} \cite{cite:PAMI14CMNN}.

%\begin{table}[tbp]
%    \addtolength{\tabcolsep}{6pt}
%    \centering
%    \caption{The Statistics of Two Datasets}
%    \label{table:datasets}
%    \small
%    \begin{tabular}{c|c|c}
%        \Xhline{1.0pt}
%        {Dataset} & {NUS-WIDE} & {Flickr}\\
%        \hline
%        Complete Set & 195,834 & 25,000 \\
%        Database & 191,834 & 17,000 \\
%        Training Set & 10,000 & 5,000 \\
%        Validation Set & 2,000 & 1,000 \\
%        Query Set & 2,000 & 2,000 \\
%        \Xhline{1.0pt}
%    \end{tabular}
%    \normalsize
%\end{table}

%\subsection{Comparison Methods} 
%
%Comparison methods include 7 state-of-the-art cross-modal hashing methods: \textbf{IMH}\footnote{\scriptsize\url{http://staff.itee.uq.edu.au/shenht/UQ_IMH}} \cite{cite:SIGMOD13IMH}, \textbf{CVH}\footref{foot:CMSSH} \cite{cite:IJCAI11CVH}, \textbf{CMSSH}\footnote{\scriptsize\url{http://www.cse.ust.hk/~dyyeung/code/mlbe.zip}\label{foot:CMSSH}} \cite{cite:CVPR10CMSSH}, \textbf{SCM}\footnote{\scriptsize{Since the original implementation is not publicly available, we implement it by ourselves.}\label{foot:SCM}}\cite{cite:AAAI14SCM}, \textbf{QCH}\footref{foot:SCM}\cite{cite:IJCAI15QCH},  \textbf{CorrAE}\footref{foot:SCM} \cite{cite:MM14CorrAE} and \textbf{CM-NN}\footref{foot:SCM}\cite{cite:PAMI14CMNN}, which can be divided into three types:
%\begin{itemize}
%    \setlength\itemsep{0pt}
%    \item \textbf{Shallow + Unsupervised.} \textbf{IMH} embeds intra-media and cross-media similarities into a common Hamming space, and integrates a linear regression to learn hash functions such that hash codes for new data can be efficiently generated. \textbf{CVH} extends spectral hashing \cite{cite:NIPS09SH} to multimodal scenario by mapping similar objects across different modalities to similar binary codes.
%    \item \textbf{Shallow + Supervised.} \textbf{CMSSH} is a pioneering work which embeds multimodal data into a common Hamming space via supervised similarity learning. \textbf{SCM} is a relevant work which seamlessly integrates semantic labels into the hashing learning procedure for the large-scale data modeling. \textbf{QCH} is the most recent work which simultaneously optimizes the quantization and maximizes the domain correlation.
%    \item \textbf{Deep.} \textbf{CorrAE} learns a multimodal auto-encoder by incorporating representation learning and correlation learning into a single process. \textbf{CM-NN} is a most recent deep learning approach based on a novel coupled siamese neural network architecture.
%\end{itemize}

%\subsection{Implementation Details}
%
%We adopt \emph{Mean Average Precision} (MAP) to measure the effectiveness of cross-modal search which is widely used in\cite{cite:SIGMOD13IMH,cite:MM13LCMH,cite:VLDB14MSAE,cite:SIGIR14DCDH,cite:MM14CorrAE}. Given a query and a set of $R$ retrieved documents, \emph{Average Precision} (AP) is defined as
%\begin{equation}
%    {\text{AP}}\text{@}R = \frac{{\sum\nolimits_{r = 1}^R {P\left( r \right)\delta \left( r \right)} }}{{\sum\nolimits_{r' = 1}^R {\delta \left( {r'} \right)} }},
%\end{equation}
%where $P(r)$ denotes the precision of the top $r$ retrieved results, and $\delta(r) = 1$ if the $r$-th retrieved result is a true neighbor of the query, otherwise $\delta(r) = 0$, and here we follow literatures \cite{cite:SIGIR13CMR,cite:SIGIR14DCDH,cite:VLDB14MSAE} to adopt AP@$R=50$. Then MAP is computed as the mean of all the queries' average precision.
%
We follow \cite{cite:PAMI14CMNN,cite:MM14CorrAE,cite:AAAI14SCM,cite:IJCAI15QCH} to evaluate the retrieval quality based on three widely used metrics: Mean Average Precision(MAP), \emph{precision-recall} curves \cite{cite:MM13LCMH,cite:VLDB14MSAE} and \emph{precision@top-R} curves.
We adopt MAP@$R=50$ following previous work \cite{cite:SIGIR13CMR,cite:SIGIR14DCDH,cite:VLDB14MSAE}.

We implement the CHN model based on the open-source \textbf{Caffe} framework \cite{cite:MM14Caffe}. 
For image network, we employ the AlexNet architecture \cite{cite:NIPS12CNN}, fine-tune convolutional layers $conv1$--$conv5$ and fully-connected layers $fc6$--$fc7$ that were copied from the pre-trained model, and train hashing layer $fch$, all via back-propagation. As the $fch$ layer is trained from scratch, we set its learning rate to be 10 times that of the lower layers. 
For text network, we employ a three layer multi-layer perceptrons (MLP), in which the $fc6$--$fc7$ layers have 256 ReLU units with dropout rate 0.5, and the $fch$ layer have $b$ $\rm{tanh}$ units.
We use the mini-batch stochastic gradient descent (SGD) with 0.9 momentum and the learning rate annealing strategy implemented in Caffe, cross-validate learning rate from $10^{-5}$ to $1$ with a multiplicative step-size $10$, 
%It's worth noting that the learning rates of image network and text network are different.
%, which are proportional to the single-modality pre-trained network, respectively.
and fix mini-batch size as $64$.

The CHN approach involves two model parameters, coefficient $\lambda$ and $\gamma$, the relative weights of the within-modal cosine loss and the cosine quantization loss. We  automatically select $\lambda$ and $\gamma$ using cross-validation.
For the comparison methods, we use cross-validation to carefully tune their parameters, respectively. Each experiment repeats ten runs and the average results are reported.

\subsection{Results and Discussions}
We compare CHN with seven state-of-the-art methods in terms of MAP, precision-recall and precision@top-R curves of two cross-modal retrieval tasks: image query on text database ($ I \rightarrow T$), text query on image database ($ T \rightarrow I$).

We evaluate all methods with different lengths of hash codes, i.e. 8, 16, 32, and 64 bits, and report their MAP results in Table \ref{table:MAP}. 
From the experimental results, we can observe that CHN substantially outperforms all state-of-the-art methods for all cross-modal tasks on the two benchmark datasets. Specifically, compared to the best shallow baseline QCH, we achieve absolute increases of 28.70\% / 22.04\% and 17.71\% / 13.08\% in average MAP for two cross-modal tasks $ I \rightarrow T$ / $ T \rightarrow I$ on NUS-WIDE and Flickr, respectively. CHN outperforms the state-of-the-art deep hashing method CM-NN by 15.22\% / 13.46\%  and 10.16\% / 5.45\% in average MAP for two cross-modal tasks on the datasets.
%b) To attain a deeper insight into the quantization loss, the MAP results of the variant CHN-Q are also shown in Table \ref{table:MAP}, which can be observed that CHN performs better than CHN-Q in both two tasks with 0.59\% / 0.82\% and 0.59\% / 0.82\% increments for NUS-WIDE and Flickr datasets, showing the effectiveness of quantization.

The precision-recall curves with 32 bits for two cross-modal tasks $ I \rightarrow T$ and $ T \rightarrow I$ on two benchmark datasets NUS-WIDE and Flickr are illustrated in Figure \ref{fig:pr}, respectively. CHN shows the best cross-modal retrieval performance at all recall levels. This validates that CHN is robust to diverse retrieval scenarios, such as higher recall tolerating lower precision. 
Figure \ref{fig:prec} shows the precision@top-R curves of all state-of-the-art methods, which further represent the precision changes along with the number of top retrieved results R with 32 bits on NUS-WIDE and Flickr datasets, respectively. We can easily observe that CHN significantly outperforms all of the state-of-the-art methods, which shows that CHN is suitable for the applications that prefer higher precision with fewer top-R retrieved results.
%The precision@top-R curves @ 16 and 32 bits for two cross-modal tasks on two benchmark datasets, NUS-WIDE and Flickr, are illustrated in Figure \ref{fig:pr}

\subsection{Empirical Analysis}\label{section:Empirical}

In order to study the effectiveness of our approach, we investigate several variants of CHN: CHN-W is the CHN variant with only within-modal squared cosine loss ${C}_{xx}$ and $C_{yy}$ \eqref{eqn:IntraL}; CHN-C is the CHN variant  with only the cross-modal squared cosine loss $C_{xy}$ \eqref{eqn:CrossL}; CHN-Q is a stronger baseline with both within-modal and cross-modal squared cosine loss, but without the cosine quantization loss \eqref{eqn:QuanL}. We report the MAP results of all CHN variants in Table~\ref{table:EmpirMAP}.

From Table~\ref{table:EmpirMAP}, we may have the following observations: (1) CHN-Q outperforms CHN-W for very large margins of 39.93\% / 38.71\% and 22.45\% / 28.13\% in average MAP on two cross-modal tasks on two benchmark datasets, confirming the vital importance of maximizing the cross-modal correlation. (2) CHN-Q also significantly outperforms CHN-C by 6.11\% / 9.52\% and 1.83\% / 2.03\% in average MAP, confirming that by preserving the within-modal correlation, the network can better exploit modality-specific knowledge structure for learning more powerful deep representations. (3) Compared to CHN-Q, the MAP results of CHN demonstrate that the cosine quantization loss can evidently reduce the loss when binarizing the continuous representations to hash codes and achieve better cross-modal retrieval results.

\section{Conclusion}\label{section:Conclusion}
In this paper, we proposed a Correlation Hashing Network (CHN) for cross-modal retrieval. CHN is a hybrid architecture that jointly optimizes the squared cosine loss on semantic similarity pairs and the cosine quantization loss on compact hash codes. Extensive experiments on standard cross-media retrieval datasets show the CHN model yields substantial boosts over the state-of-the-art hashing methods.

\newpage
\begin{small}
    \bibliographystyle{ieee}
    \bibliography{CDH}
\end{small}


\end{document}

CDH.bbl
\begin{thebibliography}{10}\itemsep=-1pt

\bibitem{cite:ICML13DCCA}
G.~Andrew, R.~Arora, J.~Bilmes, and K.~Livescu.
\newblock Deep canonical correlation analysis.
\newblock In {\em Proc. 30th ICML}, 2013.

\bibitem{cite:TPAMI13DL}
Y.~Bengio, A.~Courville, and P.~Vincent.
\newblock Representation learning: A review and new perspectives.
\newblock {\em TPAMI}, 35, 2013.

\bibitem{cite:CVPR10CMSSH}
M.~Bronstein, A.~Bronstein, F.~Michel, and N.~Paragios.
\newblock Data fusion through cross-modality metric learning using
  similarity-sensitive hashing.
\newblock In {\em CVPR}. IEEE, 2010.

\bibitem{cite:CIVR09NusWide}
T.-S. Chua, J.~Tang, R.~Hong, H.~Li, Z.~Luo, and Y.-T. Zheng.
\newblock Nus-wide: A real-world web image database from national university of
  singapore.
\newblock In {\em CIVR}. ACM, 2009.

\bibitem{cite:CVPR15RNN}
J.~Donahue, L.~A. Hendricks, S.~Guadarrama, M.~Rohrbach, S.~Venugopalan,
  T.~Darrell, and K.~Saenko.
\newblock Long-term recurrent convolutional networks for visual recognition and
  description.
\newblock In {\em {CVPR} 2015}, 2015.

\bibitem{cite:MM14CorrAE}
F.~Feng, X.~Wang, and R.~Li.
\newblock Cross-modal retrieval with correspondence autoencoder.
\newblock In {\em MM}. ACM, 2014.

\bibitem{cite:CVPR11ITQ}
Y.~Gong and S.~Lazebnik.
\newblock Iterative quantization: A procrustean approach to learning binary
  codes.
\newblock In {\em CVPR 2011}. IEEE, 2011.

\bibitem{cite:MM14IMVH}
Y.~Hu, Z.~Jin, H.~Ren, D.~Cai, and X.~He.
\newblock Iterative multi-view hashing for cross media indexing.
\newblock In {\em MM}. ACM, 2014.

\bibitem{cite:ICMR08Flickr1M}
M.~J. Huiskes and M.~S. Lew.
\newblock The mir flickr retrieval evaluation.
\newblock In {\em ICMR}. ACM, 2008.

\bibitem{cite:MM14Caffe}
Y.~Jia, E.~Shelhamer, J.~Donahue, S.~Karayev, J.~Long, R.~Girshick,
  S.~Guadarrama, and T.~Darrell.
\newblock Caffe: Convolutional architecture for fast feature embedding.
\newblock In {\em ACM Multimedia Conference}. ACM, 2014.

\bibitem{cite:NIPS12CNN}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In {\em Advances in Neural Information Processing Systems (NIPS)},
  2012.

\bibitem{cite:IJCAI11CVH}
S.~Kumar and R.~Udupa.
\newblock Learning hash functions for cross-view similarity search.
\newblock In {\em IJCAI}, 2011.

\bibitem{cite:CVPR15DNNH}
H.~Lai, Y.~Pan, Y.~Liu, and S.~Yan.
\newblock Simultaneous feature learning and hash coding with deep neural
  networks.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR)}. IEEE, 2015.

\bibitem{cite:Doc2Vec}
Q.~V. Le and T.~Mikolov.
\newblock Distributed representations of sentences and documents.
\newblock In {\em Advances in neural information processing systems}, 2014.

\bibitem{cite:ICLR14NIN}
M.~Lin, Q.~Chen, and S.~Yan.
\newblock Network in network.
\newblock In {\em International Conference on Learning Representations (ICLR),
  2014 (arXiv:1409.1556)}, 2014.

\bibitem{cite:CVPR12KSH}
W.~Liu, J.~Wang, R.~Ji, Y.-G. Jiang, and S.-F. Chang.
\newblock Supervised hashing with kernels.
\newblock In {\em IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}. IEEE, 2012.

\bibitem{cite:CVPR14CH}
X.~Liu, J.~He, C.~Deng, and B.~Lang.
\newblock Collaborative hashing.
\newblock In {\em CVPR}. IEEE, 2014.

\bibitem{cite:SIGIR13CMR}
X.~Lu, F.~Wu, S.~Tang, Z.~Zhang, X.~He, and Y.~Zhuang.
\newblock A low rank structural large margin method for cross-modal ranking.
\newblock In {\em SIGIR}. ACM, 2013.

\bibitem{cite:PAMI14CMNN}
J.~Masci, M.~M. Bronstein, A.~M. Bronstein, and J.~Schmidhuber.
\newblock Multimodal similarity-preserving hashing.
\newblock {\em {IEEE} Trans. Pattern Anal. Mach. Intell.}, 36, 2014.

\bibitem{cite:NIPS13Word2Vec}
T.~Mikolov, I.~Sutskever, K.~Chen, G.~S. Corrado, and J.~Dean.
\newblock Distributed representations of words and phrases and their
  compositionality.
\newblock In {\em Advances in neural information processing systems}, 2013.

\bibitem{cite:TPAMI14Wiki}
J.~C. Pereira, E.~Coviello, G.~Doyle, N.~Rasiwasia, G.~R.~G. Lanckriet,
  R.~Levy, and N.~Vasconcelos.
\newblock On the role of correlation and abstraction in cross-modal multimedia
  retrieval.
\newblock {\em TPAMI}, 36, 2014.

\bibitem{cite:MIT86MLP}
D.~E. Rumelhart, G.~E. Hinton, and R.~J. Williams.
\newblock Parallel distributed processing: Explorations in the microstructure
  of cognition, vol. 1.
\newblock chapter Learning Internal Representations by Error Propagation. MIT
  Press, 1986.

\bibitem{cite:TPAMI00SemanticGap}
A.~W. Smeulders, M.~Worring, S.~Santini, A.~Gupta, and R.~Jain.
\newblock Content-based image retrieval at the end of the early years.
\newblock {\em TPAMI}, 22, 2000.

\bibitem{cite:SIGMOD13IMH}
J.~Song, Y.~Yang, Y.~Yang, Z.~Huang, and H.~T. Shen.
\newblock Inter-media hashing for large-scale retrieval from heterogeneous data
  sources.
\newblock In {\em SIGMOD}. ACM, 2013.

\bibitem{cite:JMLR14MMDL}
N.~Srivastava and R.~Salakhutdinov.
\newblock Multimodal learning with deep boltzmann machines.
\newblock {\em JMLR}, 15, 2014.

\bibitem{cite:MM14DL}
J.~Wan, D.~Wang, S.~C.~H. Hoi, P.~Wu, J.~Zhu, Y.~Zhang, and J.~Li.
\newblock Deep learning for content-based image retrieval: A comprehensive
  study.
\newblock In {\em MM}. ACM, 2014.

\bibitem{cite:arxiv14hashsurvey}
J.~Wang, H.~T. Shen, J.~Song, and J.~Ji.
\newblock Hashing for similarity search: A survey.
\newblock Arxiv, 2014.

\bibitem{cite:VLDB14MSAE}
W.~Wang, B.~C. Ooi, X.~Yang, D.~Zhang, and Y.~Zhuang.
\newblock Effective multi-modal retrieval based on stacked auto-encoders.
\newblock In {\em VLDB}. ACM, 2014.

\bibitem{cite:NIPS09SH}
Y.~Weiss, A.~Torralba, and R.~Fergus.
\newblock Spectral hashing.
\newblock In {\em NIPS}, 2009.

\bibitem{cite:IJCAI15QCH}
B.~Wu, Q.~Yang, W.~Zheng, Y.~Wang, and J.~Wang.
\newblock Quantized correlation hashing for fast cross-modal search.
\newblock In {\em Proceedings of the Twenty-Fourth International Joint
  Conference on Artificial Intelligence, {IJCAI} 2015, Buenos Aires, Argentina,
  July 25-31, 2015}, 2015.

\bibitem{cite:AAAI14CNNH}
R.~Xia, Y.~Pan, H.~Lai, C.~Liu, and S.~Yan.
\newblock Supervised hashing for image retrieval via image representation
  learning.
\newblock In {\em Proceedings of the AAAI Conference on Artificial Intellignece
  (AAAI)}. AAAI, 2014.

\bibitem{cite:SIGIR14DCDH}
Z.~Yu, F.~Wu, Y.~Yang, Q.~Tian, J.~Luo, and Y.~Zhuang.
\newblock Discriminative coupled dictionary hashing for fast cross-media
  retrieval.
\newblock In {\em SIGIR}. ACM, 2014.

\bibitem{cite:AAAI14SCM}
D.~Zhang and W.~Li.
\newblock Large-scale supervised multimodal hashing with semantic correlation
  maximization.
\newblock In {\em AAAI}, 2014.

\bibitem{cite:NIPS12CRH}
Y.~Zhen and D.-Y. Yeung.
\newblock Co-regularized hashing for multimodal data.
\newblock In {\em NIPS}, 2012.

\bibitem{cite:KDD12MLBE}
Y.~Zhen and D.-Y. Yeung.
\newblock A probabilistic model for multimodal hash function learning.
\newblock In {\em SIGKDD}. ACM, 2012.

\bibitem{cite:MM13LCMH}
X.~Zhu, Z.~Huang, H.~T. Shen, and X.~Zhao.
\newblock Linear cross-modal hashing for efficient multimedia search.
\newblock In {\em MM}. ACM, 2013.

\end{thebibliography}
