  Video Question Answering (VideoQA) is a challenging video understanding task
since it requires a deep understanding of both question and video. Previous
studies mainly focus on extracting sophisticated visual and language
embeddings, fusing them by delicate hand-crafted networks. However, the
relevance of different frames, objects, and modalities to the question are
varied along with the time, which is ignored in most of existing methods.
Lacking understanding of the the dynamic relationships and interactions among
objects brings a great challenge to VideoQA task. To address this problem, we
propose a novel Relation-aware Hierarchical Attention (RHA) framework to learn
both the static and dynamic relations of the objects in videos. In particular,
videos and questions are embedded by pre-trained models firstly to obtain the
visual and textual features. Then a graph-based relation encoder is utilized to
extract the static relationship between visual objects. To capture the dynamic
changes of multimodal objects in different video frames, we consider the
temporal, spatial, and semantic relations, and fuse the multimodal features by
hierarchical attention mechanism to predict the answer. We conduct extensive
experiments on a large scale VideoQA dataset, and the experimental results
demonstrate that our RHA outperforms the state-of-the-art methods.
