  Egocentric action recognition is gaining significant attention in the field
of human action recognition. In this paper, we address data scarcity issue in
egocentric action recognition from a compositional generalization perspective.
To tackle this problem, we propose a free-form composition network (FFCN) that
can simultaneously learn disentangled verb, preposition, and noun
representations, and then use them to compose new samples in the feature space
for rare classes of action videos. First, we use a graph to capture the
spatial-temporal relations among different hand/object instances in each action
video. We thus decompose each action into a set of verb and preposition
spatial-temporal representations using the edge features in the graph. The
temporal decomposition extracts verb and preposition representations from
different video frames, while the spatial decomposition adaptively learns verb
and preposition representations from action-related instances in each frame.
With these spatial-temporal representations of verbs and prepositions, we can
compose new samples for those rare classes in a free-form manner, which is not
restricted to a rigid form of a verb and a noun. The proposed FFCN can directly
generate new training data samples for rare classes, hence significantly
improve action recognition performance. We evaluated our method on three
popular egocentric action recognition datasets, Something-Something V2, H2O,
and EPIC-KITCHENS-100, and the experimental results demonstrate the
effectiveness of the proposed method for handling data scarcity problems,
including long-tailed and few-shot egocentric action recognition.
