  Understanding dynamic hand motions and actions from egocentric RGB videos is
a fundamental yet challenging task due to self-occlusion and ambiguity. To
address occlusion and ambiguity, we develop a transformer-based framework to
exploit temporal information for robust estimation. Noticing the different
temporal granularity of and the semantic correlation between hand pose
estimation and action recognition, we build a network hierarchy with two
cascaded transformer encoders, where the first one exploits the short-term
temporal cue for hand pose estimation, and the latter aggregates per-frame pose
and object information over a longer time span to recognize the action. Our
approach achieves competitive results on two first-person hand action
benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design
choices.
