  Nuanced understanding and the generation of detailed descriptive content for
(bimanual) manipulation actions in videos is important for disciplines such as
robotics, human-computer interaction, and video content analysis. This study
describes a novel method, integrating graph based modeling with layered
hierarchical attention mechanisms, resulting in higher precision and better
comprehensiveness of video descriptions. To achieve this, we encode, first, the
spatio-temporal inter dependencies between objects and actions with scene
graphs and we combine this, in a second step, with a novel 3-level architecture
creating a hierarchical attention mechanism using Graph Attention Networks
(GATs). The 3-level GAT architecture allows recognizing local, but also global
contextual elements. This way several descriptions with different semantic
complexity can be generated in parallel for the same video clip, enhancing the
discriminative accuracy of action recognition and action description. The
performance of our approach is empirically tested using several 2D and 3D
datasets. By comparing our method to the state of the art we consistently
obtain better performance concerning accuracy, precision, and contextual
relevance when evaluating action recognition as well as description generation.
In a large set of ablation experiments we also assess the role of the different
components of our model. With our multi-level approach the system obtains
different semantic description depths, often observed in descriptions made by
different people, too. Furthermore, better insight into bimanual hand-object
interactions as achieved by our model may portend advancements in the field of
robotics, enabling the emulation of intricate human actions with heightened
precision.
