  To understand a complex action, multiple sources of information, including
appearance, positional, and semantic features, need to be integrated. However,
these features are difficult to be fused since they often differ significantly
in modality and dimensionality. In this paper, we present a novel framework
that accomplishes this goal by interactive fusion, namely, projecting features
across different spaces and guiding it using an auxiliary prediction task.
Specifically, we implement the framework in three steps, namely,
positional-to-appearance feature extraction, semantic feature interaction, and
semantic-to-positional prediction. We evaluate our approach on two action
recognition datasets, Something-Something and Charades. Interactive fusion
achieves consistent accuracy gain beyond off-the-shelf action recognition
algorithms. In particular, on Something-Else, the compositional setting of
Something-Something, interactive fusion reports a remarkable gain of 2.9% in
terms of top-1 accuracy.
