  To overcome the barrier of storage and computation, the hashing technique has
been widely used for nearest neighbor search in multimedia retrieval
applications recently. Particularly, cross-modal retrieval that searches across
different modalities becomes an active but challenging problem. Although dozens
of cross-modal hashing algorithms are proposed to yield compact binary codes,
the exhaustive search is impractical for the real-time purpose, and Hamming
distance computation suffers inaccurate results. In this paper, we propose a
novel search method that utilizes a probability-based index scheme over binary
hash codes in cross-modal retrieval. The proposed hash code indexing scheme
exploits a few binary bits of the hash code as the index code. We construct an
inverted index table based on index codes and train a neural network to improve
the indexing accuracy and efficiency. Experiments are performed on two
benchmark datasets for retrieval across image and text modalities, where hash
codes are generated by three cross-modal hashing methods. Results show the
proposed method effectively boost the performance on these hash methods.
