  This paper strives to find the sentence best describing the content of an
image or video. Different from existing works, which rely on a joint subspace
for image / video to sentence matching, we propose to do so in a visual space
only. We contribute Word2VisualVec, a deep neural network architecture that
learns to predict a deep visual encoding of textual input based on sentence
vectorization and a multi-layer perceptron. We thoroughly analyze its
architectural design, by varying the sentence vectorization strategy, network
depth and the deep feature to predict for image to sentence matching. We also
generalize Word2VisualVec for matching a video to a sentence, by extending the
predictive abilities to 3-D ConvNet features as well as a visual-audio
representation. Experiments on four challenging image and video benchmarks
detail Word2VisualVec's properties, capabilities for image and video to
sentence matching, and on all datasets its state-of-the-art results.
