  Long-term complex activity recognition and localisation can be crucial for
decision making in autonomous systems such as smart cars and surgical robots.
Here we address the problem via a novel deformable, spatiotemporal scene graph
approach, consisting of three main building blocks: (i) action tube detection,
(ii) the modelling of the deformable geometry of parts, and (iii) a graph
convolutional network. Firstly, action tubes are detected in a series of
snippets. Next, a new 3D deformable RoI pooling layer is designed for learning
the flexible, deformable geometry of the constituent action tubes. Finally, a
scene graph is constructed by considering all parts as nodes and connecting
them based on different semantics such as order of appearance, sharing the same
action label and feature similarity. We also contribute fresh temporal complex
activity annotation for the recently released ROAD autonomous driving and
SARAS-ESAD surgical action datasets and show the adaptability of our framework
to different domains. Our method is shown to significantly outperform
graph-based competitors on both augmented datasets.
