  For a given video-based Human-Object Interaction scene, modeling the
spatio-temporal relationship between humans and objects are the important cue
to understand the contextual information presented in the video. With the
effective spatio-temporal relationship modeling, it is possible not only to
uncover contextual information in each frame but also to directly capture
inter-time dependencies. It is more critical to capture the position changes of
human and objects over the spatio-temporal dimension when their appearance
features may not show up significant changes over time. The full use of
appearance features, the spatial location and the semantic information are also
the key to improve the video-based Human-Object Interaction recognition
performance. In this paper, Spatio-Temporal Interaction Graph Parsing Networks
(STIGPN) are constructed, which encode the videos with a graph composed of
human and object nodes. These nodes are connected by two types of relations:
(i) spatial relations modeling the interactions between human and the
interacted objects within each frame. (ii) inter-time relations capturing the
long range dependencies between human and the interacted objects across frame.
With the graph, STIGPN learn spatio-temporal features directly from the whole
video-based Human-Object Interaction scenes. Multi-modal features and a
multi-stream fusion strategy are used to enhance the reasoning capability of
STIGPN. Two Human-Object Interaction video datasets, including CAD-120 and
Something-Else, are used to evaluate the proposed architectures, and the
state-of-the-art performance demonstrates the superiority of STIGPN.
