  Audio and vision are two main modalities in video data. Multimodal learning,
especially for audiovisual learning, has drawn considerable attention recently,
which can boost the performance of various computer vision tasks. However, in
video summarization, existing approaches just exploit the visual information
while neglect the audio information. In this paper, we argue that the audio
modality can assist vision modality to better understand the video content and
structure, and further benefit the summarization process. Motivated by this, we
propose to jointly exploit the audio and visual information for the video
summarization task, and develop an AudioVisual Recurrent Network (AVRN) to
achieve this. Specifically, the proposed AVRN can be separated into three
parts: 1) the two-stream LSTM is utilized to encode the audio and visual
feature sequentially by capturing their temporal dependency. 2) the audiovisual
fusion LSTM is employed to fuse the two modalities by exploring the latent
consistency between them. 3) the self-attention video encoder is adopted to
capture the global dependency in the video. Finally, the fused audiovisual
information, and the integrated temporal and global dependencies are jointly
used to predict the video summary. Practically, the experimental results on the
two benchmarks, \emph{i.e.,} SumMe and TVsum, have demonstrated the
effectiveness of each part, and the superiority of AVRN compared to those
approaches just exploiting visual information for video summarization.
