  We present a general framework for compositional action recognition -- i.e.
action recognition where the labels are composed out of simpler components such
as subjects, atomic-actions and objects. The main challenge in compositional
action recognition is that there is a combinatorially large set of possible
actions that can be composed using basic components. However, compositionality
also provides a structure that can be exploited. To do so, we develop and test
a novel Structured Attention Fusion (SAF) self-attention mechanism to combine
information from object detections, which capture the time-series structure of
an action, with visual cues that capture contextual information. We show that
our approach recognizes novel verb-noun compositions more effectively than
current state of the art systems, and it generalizes to unseen action
categories quite efficiently from only a few labeled examples. We validate our
approach on the challenging Something-Else tasks from the
Something-Something-V2 dataset. We further show that our framework is flexible
and can generalize to a new domain by showing competitive results on the
Charades-Fewshot dataset.
