  Recently, automatic video captioning has attracted increasing attention,
where the core challenge lies in capturing the key semantic items, like objects
and actions as well as their spatial-temporal correlations from the redundant
frames and semantic content. To this end, existing works select either the key
video clips in a global level~(across multi frames), or key regions within each
frame, which, however, neglect the hierarchical order, i.e., key frames first
and key regions latter. In this paper, we propose a novel joint-hierarchical
attention model for video captioning, which embeds the key clips, the key
frames and the key regions jointly into the captioning model in a hierarchical
manner. Such a joint-hierarchical attention model first conducts a global
selection to identify key frames, followed by a Gumbel sampling operation to
identify further key regions based on the key frames, achieving an accurate
global-to-local feature representation to guide the captioning. Extensive
quantitative evaluations on two public benchmark datasets MSVD and MSR-VTT
demonstrates the superiority of the proposed method over the state-of-the-art
methods.
