  Although video summarization has achieved tremendous success benefiting from
Recurrent Neural Networks (RNN), RNN-based methods neglect the global
dependencies and multi-hop relationships among video frames, which limits the
performance. Transformer is an effective model to deal with this problem, and
surpasses RNN-based methods in several sequence modeling tasks, such as machine
translation, video captioning, \emph{etc}. Motivated by the great success of
transformer and the natural structure of video (frame-shot-video), a
hierarchical transformer is developed for video summarization, which can
capture the dependencies among frame and shots, and summarize the video by
exploiting the scene information formed by shots. Furthermore, we argue that
both the audio and visual information are essential for the video summarization
task. To integrate the two kinds of information, they are encoded in a
two-stream scheme, and a multimodal fusion mechanism is developed based on the
hierarchical transformer. In this paper, the proposed method is denoted as
Hierarchical Multimodal Transformer (HMT). Practically, extensive experiments
show that HMT surpasses most of the traditional, RNN-based and attention-based
video summarization methods.
