  Actions are about how we interact with the environment, including other
people, objects, and ourselves. In this paper, we propose a novel multi-modal
Holistic Interaction Transformer Network (HIT) that leverages the largely
ignored, but critical hand and pose information essential to most human
actions. The proposed "HIT" network is a comprehensive bi-modal framework that
comprises an RGB stream and a pose stream. Each of them separately models
person, object, and hand interactions. Within each sub-network, an
Intra-Modality Aggregation module (IMA) is introduced that selectively merges
individual interaction units. The resulting features from each modality are
then glued using an Attentive Fusion Mechanism (AFM). Finally, we extract cues
from the temporal context to better classify the occurring actions using cached
memory. Our method significantly outperforms previous approaches on the J-HMDB,
UCF101-24, and MultiSports datasets. We also achieve competitive results on
AVA. The code will be available at https://github.com/joslefaure/HIT.
