  Egocentric temporal action segmentation in videos is a crucial task in
computer vision with applications in various fields such as mixed reality,
human behavior analysis, and robotics. Although recent research has utilized
advanced visual-language frameworks, transformers remain the backbone of action
segmentation models. Therefore, it is necessary to improve transformers to
enhance the robustness of action segmentation models. In this work, we propose
two novel ideas to enhance the state-of-the-art transformer for action
segmentation. First, we introduce a dual dilated attention mechanism to
adaptively capture hierarchical representations in both local-to-global and
global-to-local contexts. Second, we incorporate cross-connections between the
encoder and decoder blocks to prevent the loss of local context by the decoder.
We also utilize state-of-the-art visual-language representation learning
techniques to extract richer and more compact features for our transformer. Our
proposed approach outperforms other state-of-the-art methods on the Georgia
Tech Egocentric Activities (GTEA) and HOI4D Office Tools datasets, and we
validate our introduced components with ablation studies. The source code and
supplementary materials are publicly available on
https://www.sail-nu.com/dxformer.
