  Exploiting the inner-shot and inter-shot dependencies is essential for
key-shot based video summarization. Current approaches mainly devote to
modeling the video as a frame sequence by recurrent neural networks. However,
one potential limitation of the sequence models is that they focus on capturing
local neighborhood dependencies while the high-order dependencies in long
distance are not fully exploited. In general, the frames in each shot record a
certain activity and vary smoothly over time, but the multi-hop relationships
occur frequently among shots. In this case, both the local and global
dependencies are important for understanding the video content. Motivated by
this point, we propose a Reconstructive Sequence-Graph Network (RSGN) to encode
the frames and shots as sequence and graph hierarchically, where the
frame-level dependencies are encoded by Long Short-Term Memory (LSTM), and the
shot-level dependencies are captured by the Graph Convolutional Network (GCN).
Then, the videos are summarized by exploiting both the local and global
dependencies among shots. Besides, a reconstructor is developed to reward the
summary generator, so that the generator can be optimized in an unsupervised
manner, which can avert the lack of annotated data in video summarization.
Furthermore, under the guidance of reconstruction loss, the predicted summary
can better preserve the main video content and shot-level dependencies.
Practically, the experimental results on three popular datasets i.e., SumMe,
TVsum and VTW) have demonstrated the superiority of our proposed approach to
the summarization task.
