  3D hand poses are an under-explored modality for action recognition. Poses
are compact yet informative and can greatly benefit applications with limited
compute budgets. However, poses alone offer an incomplete understanding of
actions, as they cannot fully capture objects and environments with which
humans interact. To efficiently model hand-object interactions, we propose
HandFormer, a novel multimodal transformer. HandFormer combines 3D hand poses
at a high temporal resolution for fine-grained motion modeling with sparsely
sampled RGB frames for encoding scene semantics. Observing the unique
characteristics of hand poses, we temporally factorize hand modeling and
represent each joint by its short-term trajectories. This factorized pose
representation combined with sparse RGB samples is remarkably efficient and
achieves high accuracy. Unimodal HandFormer with only hand poses outperforms
existing skeleton-based methods at 5x fewer FLOPs. With RGB, we achieve new
state-of-the-art performance on Assembly101 and H2O with significant
improvements in egocentric action recognition.
