  We present an approach that exploits hierarchical Recurrent Neural Networks
(RNNs) to tackle the video captioning problem, i.e., generating one or multiple
sentences to describe a realistic video. Our hierarchical framework contains a
sentence generator and a paragraph generator. The sentence generator produces
one simple short sentence that describes a specific short video interval. It
exploits both temporal- and spatial-attention mechanisms to selectively focus
on visual elements during generation. The paragraph generator captures the
inter-sentence dependency by taking as input the sentential embedding produced
by the sentence generator, combining it with the paragraph history, and
outputting the new initial state for the sentence generator. We evaluate our
approach on two large-scale benchmark datasets: YouTubeClips and
TACoS-MultiLevel. The experiments demonstrate that our approach significantly
outperforms the current state-of-the-art methods with BLEU@4 scores 0.499 and
0.305 respectively.
