  Humans have the natural ability to recognize actions even if the objects
involved in the action or the background are changed. Humans can abstract away
the action from the appearance of the objects and their context which is
referred to as compositionality of actions. Compositional action recognition
deals with imparting human-like compositional generalization abilities to
action-recognition models. In this regard, extracting the interactions between
humans and objects forms the basis of compositional understanding. These
interactions are not affected by the appearance biases of the objects or the
context. But the context provides additional cues about the interactions
between things and stuff. Hence we need to infuse context into the human-object
interactions for compositional action recognition. To this end, we first design
a spatial-temporal interaction encoder that captures the human-object (things)
interactions. The encoder learns the spatio-temporal interaction tokens
disentangled from the background context. The interaction tokens are then
infused with contextual information from the video tokens to model the
interactions between things and stuff. The final context-infused
spatio-temporal interaction tokens are used for compositional action
recognition. We show the effectiveness of our interaction-centric approach on
the compositional Something-Else dataset where we obtain a new state-of-the-art
result of 83.8% top-1 accuracy outperforming recent important object-centric
methods by a significant margin. Our approach of explicit human-object-stuff
interaction modeling is effective even for standard action recognition datasets
such as Something-Something-V2 and Epic-Kitchens-100 where we obtain comparable
or better performance than state-of-the-art.
