  The use of Recurrent Neural Networks for video captioning has recently gained
a lot of attention, since they can be used both to encode the input video and
to generate the corresponding description. In this paper, we present a
recurrent video encoding scheme which can discover and leverage the
hierarchical structure of the video. Unlike the classical encoder-decoder
approach, in which a video is encoded continuously by a recurrent layer, we
propose a novel LSTM cell, which can identify discontinuity points between
frames or segments and modify the temporal connections of the encoding layer
accordingly. We evaluate our approach on three large-scale datasets: the
Montreal Video Annotation dataset, the MPII Movie Description dataset and the
Microsoft Video Description Corpus. Experiments show that our approach can
discover appropriate hierarchical representations of input videos and improve
the state of the art results on movie description datasets.
