  Action recognition has typically treated actions and activities as monolithic
events that occur in videos. However, there is evidence from Cognitive Science
and Neuroscience that people actively encode activities into consistent
hierarchical part structures. However in Computer Vision, few explorations on
representations encoding event partonomies have been made. Inspired by evidence
that the prototypical unit of an event is an action-object interaction, we
introduce Action Genome, a representation that decomposes actions into
spatio-temporal scene graphs. Action Genome captures changes between objects
and their pairwise relationships while an action occurs. It contains 10K videos
with 0.4M objects and 1.7M visual relationships annotated. With Action Genome,
we extend an existing action recognition model by incorporating scene graphs as
spatio-temporal feature banks to achieve better performance on the Charades
dataset. Next, by decomposing and learning the temporal changes in visual
relationships that result in an action, we demonstrate the utility of a
hierarchical event decomposition by enabling few-shot action recognition,
achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing
scene graph models on the new task of spatio-temporal scene graph prediction.
