  The task of video-based commonsense captioning aims to generate event-wise
captions and meanwhile provide multiple commonsense descriptions (e.g.,
attribute, effect and intention) about the underlying event in the video. Prior
works explore the commonsense captions by using separate networks for different
commonsense types, which is time-consuming and lacks mining the interaction of
different commonsense. In this paper, we propose a Hybrid Reasoning Network
(HybridNet) to endow the neural networks with the capability of semantic-level
reasoning and word-level reasoning. Firstly, we develop multi-commonsense
learning for semantic-level reasoning by jointly training different commonsense
types in a unified network, which encourages the interaction between the clues
of multiple commonsense descriptions, event-wise captions and videos. Then,
there are two steps to achieve the word-level reasoning: (1) a memory module
records the history predicted sequence from the previous generation processes;
(2) a memory-routed multi-head attention (MMHA) module updates the word-level
attention maps by incorporating the history information from the memory module
into the transformer decoder for word-level reasoning. Moreover, the multimodal
features are used to make full use of diverse knowledge for commonsense
reasoning. Experiments and abundant analysis on the large-scale
Video-to-Commonsense benchmark show that our HybridNet achieves
state-of-the-art performance compared with other methods.
