  Video Semantic Role Labeling (VidSRL) aims to detect the salient events from
given videos, by recognizing the predict-argument event structures and the
interrelationships between events. While recent endeavors have put forth
methods for VidSRL, they can be mostly subject to two key drawbacks, including
the lack of fine-grained spatial scene perception and the insufficiently
modeling of video temporality. Towards this end, this work explores a novel
holistic spatio-temporal scene graph (namely HostSG) representation based on
the existing dynamic scene graph structures, which well model both the
fine-grained spatial semantics and temporal dynamics of videos for VidSRL.
Built upon the HostSG, we present a nichetargeting VidSRL framework. A
scene-event mapping mechanism is first designed to bridge the gap between the
underlying scene structure and the high-level event semantic structure,
resulting in an overall hierarchical scene-event (termed ICE) graph structure.
We further perform iterative structure refinement to optimize the ICE graph,
such that the overall structure representation can best coincide with end task
demand. Finally, three subtask predictions of VidSRL are jointly decoded, where
the end-to-end paradigm effectively avoids error propagation. On the benchmark
dataset, our framework boosts significantly over the current best-performing
model. Further analyses are shown for a better understanding of the advances of
our methods.
