  The ability to describe images with natural language sentences is the
hallmark for image and language understanding. Such a system has wide ranging
applications such as annotating images and using natural sentences to search
for images.In this project we focus on the task of bidirectional image
retrieval: such asystem is capable of retrieving an image based on a sentence
(image search) andretrieve sentence based on an image query (image annotation).
We present asystem based on a global ranking objective function which uses a
combinationof convolutional neural networks (CNN) and multi layer perceptrons
(MLP).It takes a pair of image and sentence and processes them in different
channels,finally embedding it into a common multimodal vector space. These
embeddingsencode abstract semantic information about the two inputs and can be
comparedusing traditional information retrieval approaches. For each such pair,
the modelreturns a score which is interpretted as a similarity metric. If this
score is high,the image and sentence are likely to convey similar meaning, and
if the score is low then they are likely not to.
  The visual input is modeled via deep convolutional neural network. On
theother hand we explore three models for the textual module. The first one
isbag of words with an MLP. The second one uses n-grams (bigram, trigrams,and a
combination of trigram & skip-grams) with an MLP. The third is morespecialized
deep network specific for modeling variable length sequences (SSE).We report
comparable performance to recent work in the field, even though ouroverall
model is simpler. We also show that the training time choice of how wecan
generate our negative samples has a significant impact on performance, and can
be used to specialize the bi-directional system in one particular task.
