  Fully supervised action segmentation works on frame-wise action recognition
with dense annotations and often suffers from the over-segmentation issue.
Existing works have proposed a variety of solutions such as boundary-aware
networks, multi-stage refinement, and temporal smoothness losses. However, most
of them take advantage of frame-wise supervision, which cannot effectively
tackle the evaluation metrics with different granularities. In this paper, for
the desirable large receptive field, we first develop a novel local-global
attention mechanism with temporal pyramid dilation and temporal pyramid pooling
for efficient multi-scale attention. Then we decouple two inherent goals in
action segmentation, ie, (1) individual identification solved by frame-wise
supervision, and (2) temporal reasoning tackled by action set prediction.
Afterward, an action alignment module fuses these different granularity
predictions, leading to more accurate and smoother action segmentation. We
achieve state-of-the-art accuracy, eg, 82.8% (+2.6%) on GTEA and 74.7% (+1.2%)
on Breakfast, which demonstrates the effectiveness of our proposed method,
accompanied by extensive ablation studies. The code will be made available
later.
