  Recently, deep learning approach, especially deep Convolutional Neural
Networks (ConvNets), have achieved overwhelming accuracy with fast processing
speed for image classification. Incorporating temporal structure with deep
ConvNets for video representation becomes a fundamental problem for video
content analysis. In this paper, we propose a new approach, namely Hierarchical
Recurrent Neural Encoder (HRNE), to exploit temporal information of videos.
Compared to recent video representation inference approaches, this paper makes
the following three contributions. First, our HRNE is able to efficiently
exploit video temporal structure in a longer range by reducing the length of
input information flow, and compositing multiple consecutive inputs at a higher
level. Second, computation operations are significantly lessened while
attaining more non-linearity. Third, HRNE is able to uncover temporal
transitions between frame chunks with different granularities, i.e., it can
model the temporal transitions between frames as well as the transitions
between segments. We apply the new method to video captioning where temporal
information plays a crucial role. Experiments demonstrate that our method
outperforms the state-of-the-art on video captioning benchmarks. Notably, even
using a single network with only RGB stream as input, HRNE beats all the recent
systems which combine multiple inputs, such as RGB ConvNet plus 3D ConvNet.
