  Capturing interaction of hands with objects is important to autonomously
detect human actions from egocentric videos. In this work, we present a pyramid
video transformer with a dynamic class token generator for egocentric action
recognition. Different from previous video transformers, which use the same
static embedding as the class token for diverse inputs, we propose a dynamic
class token generator that produces a class token for each input video by
analyzing the hand-object interaction and the related motion information. The
dynamic class token can diffuse such information to the entire model by
communicating with other informative tokens in the subsequent transformer
layers. With the dynamic class token, dissimilarity between videos can be more
prominent, which helps the model distinguish various inputs. In addition,
traditional video transformers explore temporal features globally, which
requires large amounts of computation. However, egocentric videos often have a
large amount of background scene transition, which causes discontinuities
across distant frames. In this case, blindly reducing the temporal sampling
rate will risk losing crucial information. Hence, we also propose a pyramid
architecture to hierarchically process the video from short-term high rate to
long-term low rate. With the proposed architecture, we significantly reduce the
computational cost as well as the memory requirement without sacrificing from
the model performance. We perform comparisons with different baseline video
transformers on the EPIC-KITCHENS-100 and EGTEA Gaze+ datasets. Both
quantitative and qualitative results show that the proposed model can
efficiently improve the performance for egocentric action recognition.
